Matrices and Derivatives: Comradely functions

Written on January 28, 2019

NOTE: This post is automatically translated from Farsi using GPT 5.5

Now that we have reviewed the initial topics of matrix derivatives together, we can talk about derivatives of general functions defined on matrices — such as the matrix inverse and the determinant.

Let’s start with the derivative of the matrix inverse. When the inverse of a matrix is multiplied by the matrix itself, the result is an identity matrix with the same dimensions as the original matrix. We can start from this very point:

$$\begin{array}{cc}& \frac{\partial A^{-1}A}{\partial A}=\frac{\partial I}{\partial A}=0\end{array}$$

On the other hand, we also had a relation for differentiating the product of two matrix functions:

$$\frac{\partial f(A)g(A)}{\partial A}=(I\otimes f(A))\frac{\partial g(A)}{\partial A}+(g(A{)}^T\otimes I)\frac{\partial f(A)}{\partial A}$$

Substituting into the first relation, we have:

$$\frac{\partial A^{-1}A}{\partial A}=(I\otimes A^{-1})+(A^T\otimes I)\frac{\partial A^{-1}}{\partial A}=0$$

And voilà!

$$\frac{\partial A^{-1}}{\partial A}=-(A^{-T}\otimes I)(I\otimes A^{-1})=-(A^{-T}\otimes A^{-1})$$

To compute this last one, I used two relations among Kronecker products:

$$\begin{array}{cc}& (A\otimes B{)}^{-1}=(A^{-1}\otimes B^{-1})\\ & (A\otimes B)(C\otimes D)=(AC\otimes BD)\end{array}$$

Of course, when the conditions are met. That is, in the first relation, both matrices $A$ and $B$ must be invertible, since if they are not, the Kronecker product will not be invertible either; and in the second relation, the matrices $A$ and $C$ must be multipliable, and likewise $B$ and $D$. In the second case, these multiplications may not be possible, while it may still be possible to multiply the outputs of the two Kronecker products in the usual matrix sense.

The next function is the determinant function. Computing this one is somewhat more complicated than the first. To obtain the derivative of the determinant, we have to pay attention to the adjugate matrix. The relation between the determinant and the adjugate matrix is very interesting. It is enough to choose any row of the adjugate matrix and multiply it by its corresponding column in the original matrix. The result is the determinant of the original matrix. This means that whatever $i$ is (in the range from 1 to the number of columns of the original matrix):

$$|A|=A_{i,.}^{\ast }{A}_{.,i}$$

On the other hand, the inverse of a matrix can also be computed using the adjugate matrix:

$$A^{-1}=\frac{1}{|A|}{A}^{\ast }$$

With these, we can finish the job:

$$\begin{array}{cc}& \frac{\partial |A|}{\partial A_{.,i}}=A_{i,.}^{\ast }\\ & \frac{\partial |A|}{\partial A}=[A_{1,.}^{\ast },A_{2,.}^{\ast },\dots A_{1,.}^{\ast }]={\text{vec}}^T((A^{\ast }{)}^T)\end{array}$$

Finally, we put $|A|A^{-1}$ in place of $A^{\ast }$, and that’s it:

$$\frac{\partial |A|}{\partial A}={\text{vec}}^T((|A|A^{-}1{)}^T)=|A|{\text{vec}}^T(A^{-T})$$

If we consider $\text{ln}|A|$, the result becomes even nicer:

$$\frac{\partial \text{ln}|A|}{\partial A}={\text{vec}}^T(A^{-T})$$

Can you say why?

The last function we address in this section is the inner product of two matrices. The inner product of two matrices is the product of their vectorized forms:

$$<A,B>={\text{vec}}^T(A)\text{vec}(B)=\text{trace}(A^{T}B)$$

In most texts, we see this last one introduced as the inner product. The reason can be seen quite easily.

$$\begin{array}{cc}& \text{trace}(A^{T}B)=\sum _i(A^{T}B{)}_{i,i}=\sum _i{A}_{.,i}^T{B}_{.,i}\\ & \text{vec}(A)=\left[\begin{array}{c}{A}_{.,1}\\ A_{.,2}\\ ⋮\\ A_{.,m}\end{array}\right],\text{vec}(B)=\left[\begin{array}{c}{B}_{.,1}\\ B_{.,2}\\ ⋮\\ B_{.,m}\end{array}\right]\\ & {\text{vec}}^T(A)\text{vec}(B)=\sum _i{A}_{.,i}^T{B}_{.,i}\end{array}$$

If you have paid attention, the number of rows of matrix $A$ and the columns of $B$ are equal. The reason is that this inner product is defined for such matrices. In principle, $\text{trace}$ is defined on square matrices, and the output of $A^{T}B$ must be square so that $\text{trace}$ can be applied to it.

The derivative of the inner product of two matrices with respect to one of them is, as expected, the transpose of the vectorized form of the other:

$$\begin{array}{cc}& \frac{\partial \text{trace}(A^{T}B)}{\partial B}={\text{vec}}^T(A)\\ & \frac{\partial \text{trace}(A^{T}B)}{\partial A}={\text{vec}}^T(B)\end{array}$$

This is the same thing we had seen in Calculus II at university for vectors:

$$\begin{array}{cc}& \frac{\partial a^{T}b}{\partial b}=a^T\\ & \frac{\partial a^{T}b}{\partial a}=b^T\end{array}$$

After three posts about derivatives with respect to matrices, we have reached a point where we can solve a problem such as estimating the correlation matrix of a Gaussian distribution from samples. We will solve a couple of examples of these problems in the next post, as a fitting conclusion to the topic of differentiation with respect to matrices.