Matrices and Derivatives: Kronecker in the Realm of Matrices

Written on January 21, 2019

NOTE: This post is automatically translated from Farsi using GPT 5.5

It was said that when we regard derivatives with respect to matrices as derivatives with respect to their vectorized forms (the same $\text{vec}$ operator), we no longer need to wrestle with tensors and covectors and contravectors. At the same time, we need to bring the Kronecker product from that world into this one. In tensor computations, the Kronecker product creates a tensor whose dimensions equal the sum of the dimensions of its input tensors. For example, the Kronecker product of two vectors is a matrix. But in the world of matrices, the Kronecker product has a simpler form.

If we have two matrices $A$ and $B$, their Kronecker product is defined as follows:

$$A\otimes B=\left[\begin{array}{cccc}{A}_{11}B& A_{12}B& \cdots & A_{1m}B\\ A_{21}B& A_{22}B& \cdots & A_{2m}B\\ ⋮& \ddots & ⋮& \\ A_{n1}B& A_{n2}B& \cdots & A_{nm}B\end{array}\right]$$

For example:

$$\left[\begin{array}{cc}3& 4\\ 5& 6\end{array}\right]\otimes \left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}3& 0& 4& 0\\ 0& 3& 0& 4\\ 5& 0& 6& 0\\ 0& 5& 0& 6\end{array}\right]$$

This product has beautiful properties, and in the context of differentiation with respect to matrices, the following property is one of the most important:

$$\text{vec}(AXB)=(B^T\otimes A)\text{vec}(X)$$

For an example of using this relation, suppose we want to differentiate $AXB$ with respect to $X$:

$$\frac{\partial AXB}{\partial X}=\frac{\partial \text{vec}(AXB)}{\partial \text{vec}(X)}=\frac{\partial (B^T\otimes A)\text{vec}(X)}{\partial \text{vec}(X)}=(B^T\otimes A)$$

Chain derivatives and the derivative of a product of functions

In the world of scalars, there were all sorts of relationships for derivatives that got the job done. These relationships also hold in similar forms in the world of matrices. One of those relationships was the scalar product and sum rule, which I mentioned in the previous post.

It is not a bad idea to see why that relation holds:

$$\begin{array}{cc}{\frac{\partial (\alpha A+\beta B)}{\partial X}}_{ij}& =\frac{\partial \text{vec}(\alpha A+\beta B{)}_i}{\partial \text{vec}(X{)}_j}=\frac{\partial \alpha \text{vec}(A{)}_i}{\partial \text{vec}(X{)}_j}+\frac{\partial \beta \text{vec}(B{)}_i}{\partial \text{vec}(X{)}_j}\\ & =\alpha \frac{\partial \text{vec}(A{)}_i}{\partial \text{vec}(X{)}_j}+\beta \frac{\partial \text{vec}(B{)}_i}{\partial \text{vec}(X{)}_j}\\ & =\alpha {\frac{\partial \text{vec}(A)}{\partial \text{vec}(X)}}_{ij}+\beta {\frac{\partial \text{vec}(B)}{\partial \text{vec}(X)}}_{ij}\end{array}$$

In the same way, one can see that the chain derivative of functions can be computed as follows:

$$\begin{array}{cc}{\frac{\partial f(g(X))}{\partial X}}_{ij}& =\sum _k\frac{\partial \text{vec}(f(g(X)){)}_i}{\partial \text{vec}(g(X){)}_k}\frac{\partial \text{vec}(g(X){)}_k}{\partial \text{vec}(X{)}_j}\\ & ={\left(\frac{\partial f(g(X))}{\partial g(X)}\frac{\partial g(X)}{\partial X}\right)}_{ij}\end{array}$$

And more generally:

$$\begin{array}{cc}{\frac{\partial h(f(X),g(X))}{\partial X}}_{ij}& =\begin{array}{cc}& \sum _k(\frac{\partial \text{vec}(h(f(X),g(X)){)}_i}{\partial \text{vec}(f(X){)}_k}\frac{\partial \text{vec}(f(X){)}_k}{\partial \text{vec}(X{)}_j}+\\ & \frac{\partial \text{vec}(h(f(X),g(X)){)}_i}{\partial \text{vec}(g(X){)}_k}\frac{\partial \text{vec}(g(X){)}_k}{\partial \text{vec}(X{)}_j})\end{array}\\ & ={(\frac{\partial h(f(X),g(X))}{\partial f(X)}\frac{\partial f(X)}{\partial X}+\frac{\partial h(f(X),g(X))}{\partial g(X)}\frac{\partial g(X)}{\partial X})}_{ij}& \end{array}$$

Apparently, those who defined the rule of matrix multiplication as multiplying the row of the first by the column of the second laid the right foundation! Everything fits together.

But what are the relations for the product of two functions?

$$\frac{\partial f(X)g(X)}{\partial X}=?$$

If you are tired of examining the derivative element by element, you have every right to be; I am tired too. So we ask Professor Kronecker for help in finding this one. First, we consider the function $h(X,Y)=XY$. The derivative of this function with respect to $X$ and $Y$ can be computed using the Kronecker relation as follows:

$$\begin{array}{cc}& \text{vec}(XYI)=(I\otimes X)\text{vec}(Y)\Rightarrow \frac{\partial XY}{\partial Y}=(I\otimes X)\\ & \text{vec}(IXY)=(Y^T\otimes I)\text{vec}(X)\Rightarrow \frac{\partial XY}{\partial X}=(Y^T\otimes I)\end{array}$$

With this, the derivative of $h(f(X),g(X))=f(X)g(X)$ becomes:

$$\frac{\partial f(X)g(X)}{\partial X}=(I\otimes f(X))\frac{\partial g(X)}{\partial X}+(g(X{)}^T\otimes I)\frac{\partial f(X)}{\partial X}$$

The fact that I wrote the identity matrices corresponding to $f$ and $g$ with the same symbol should not make you assume that they both have the same dimensions. The matrix $I$ that is Kronecker-multiplied by $f(x)$ has dimensions equal to $g(X)$, and vice versa.

There is one small point to talk about, and then we have closed the file on this post. We talked about the $\text{vec}$ operator, but we did not say anything about the effect of transpose on this operator. For example, when we want to compute the derivative of $A^T$ with respect to $A$:

$$\frac{\partial A^T}{\partial A}=\frac{\partial \text{vec}(A^T)}{\partial \text{vec}(A)}$$

What does $\text{vec}(A^T)$ become? Does it have a relation with $\text{vec}(A)$? Well, it is obvious that I have asked a nonsensical question and that they do have a relation, but what relation?

For example, let us look at a 3 by 3 matrix:

$$\begin{array}{ccc}& \text{vec}\left(\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\end{array}\right]\right)=\left[\begin{array}{c}1\\ 4\\ 7\\ 2\\ 5\\ 8\\ 3\\ 6\\ 9\end{array}\right]& \text{vec}\left({\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\end{array}\right]}^T\right)=\left[\begin{array}{c}1\\ 2\\ 3\\ 4\\ 5\\ 6\\ 7\\ 8\\ 9\end{array}\right]\end{array}$$

The relation between these two matrices can be written like this:

$$\left[\begin{array}{ccccccccc}1& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 1& 0& 0\\ 0& 1& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 1& 0\\ 0& 0& 1& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}1\\ 2\\ 3\\ 4\\ 5\\ 6\\ 7\\ 8\\ 9\end{array}\right]=\left[\begin{array}{c}1\\ 4\\ 7\\ 2\\ 5\\ 8\\ 3\\ 6\\ 9\end{array}\right]$$

Matrices consisting of one 1 in each row are permutation matrices. Their job is to place the rows of the vector (or matrix) they are multiplied by in another order. In matrix algebra, we call the permutation matrix that turns $\text{vec}(A)$ into $\text{vec}(A^T)$ the matrix $T$. But what remained was the derivative of the Kronecker product. Let us skip how to compute this one. Just know that the derivative of the Kronecker product has a result like the following:

$$\frac{\partial A\otimes B}{\partial A}=(I_n\otimes T_{qm}\otimes I_p)(I_{mn}\otimes \text{vec}(B))$$

This time I wrote the dimensions of the identity matrices and $T$, because if we do not write them, the probability of everything getting mixed up with everything else becomes high and the stew ends up in the yogurt. The expression above assumes that $A$ has dimensions $m\times n$ and $B$ has dimensions $p\times q$.

Well, we have closed the file on Kronecker too. In the next post, we will deal with functions such as the inverse and determinant of a matrix, so that in the final post, we can solve a few sample problems using the tools we have discussed.