Matrices and Derivatives: An Introduction

Written on January 19, 2019

NOTE: This post is automatically translated from Farsi using GPT 5.5

It is obvious from everything about me how much I love mathematics. Nevertheless, it was in the second year of my master’s that I was forced to seriously go after differentiating functions with matrix inputs. In the end, it was a small 14-page booklet by a professor of agricultural economics that got my work going.

Now, after more than 12 years, differentiating this kind of function is no longer considered a serious concern. These days, toolkits like PyTorch and TensorFlow themselves take derivatives for you from the function you have built, and do a thousand other things too. Nevertheless, having knowledge of differentiating functions with matrix variables can still be useful for those who wrestle with the details of optimizing their models (at least this statement is a good excuse for me to write about it!)

Functions with Matrix Inputs

Examples of functions with matrix inputs are abundant. One of the most famous is our very own Gaussian distribution function. The multivariate Gaussian distribution function:

$$p(x|\mathrm{\Sigma },\mu )=(2\pi \text{det}\mathrm{\Sigma }{)}^{-\frac{1}{2}}\text{exp}(\frac{1}{2}(x-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x-\mu ))$$

If we have a series of samples from this distribution and want to estimate the matrix $\mathrm{\Sigma }$, we will be dealing with functions whose input is a matrix.

Other examples of these functions are various kinds of neural network models. In most types of neural networks, there are a series of weight matrices, and for optimization and finding the model parameters, we will end up dealing with functions of these very weight matrices.

To see one example through to the end, let us go back again to estimating the matrix $\mathrm{\Sigma }$. We have $n$ samples of $x$, which we denote by $x_1$ and $x_2$ up to $x_n$. We want to find a $\mathrm{\Sigma }$ that maximizes the probability of occurrence of these samples. We also assume that the $x_i$s are all IID; that is, they all have the same distribution and are also independent of one another. With these explanations:

$$\begin{array}{cc}J(\mathrm{\Sigma })=p(x_1,x_2,\dots x_n|\mathrm{\Sigma },\mu )& =\prod (2\pi \text{det}\mathrm{\Sigma }{)}^{-\frac{1}{2}}\text{exp}(\frac{1}{2}(x_i-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x_i-\mu ))\\ & =(2\pi \text{det}\mathrm{\Sigma }{)}^{-\frac{n}{2}}\text{exp}(\frac{1}{2}\sum (x_i-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x_i-\mu ))\end{array}$$

is the function that must be maximized with respect to its input, namely $\mathrm{\Sigma }$. A function with a matrix input from which we must take the derivative with respect to the input, set it equal to zero, and find the optimum value.

Differentiating Matrix Functions Without Pain and Bloodshed

If we want to differentiate functions with matrix inputs in a principled way, we will end up dealing with tensors and tensor operators. But there is also a simpler way. This simple way is based on a neat perspective: “When computing a derivative, the arrangement of the elements of a matrix is a matter of secondary importance.”

When we approach differentiation with respect to matrices with this view, we no longer pay heed to the arrangement of the elements of the matrix! Every matrix is a large vector together with two numbers that represent how the elements are arranged. For example:

$$\left[\begin{array}{cc}5& 4\\ 3& 2\\ 1& 0\end{array}\right]\sim (\left[\begin{array}{c}5\\ 3\\ 1\\ 4\\ 2\\ 0\end{array}\right],(\mathrm{3,2}))$$

If you pay attention, in this new representation nothing has really been lost. We can reconstruct the first matrix again. We have a vector that is the columns of the original matrix glued together, and an ordered pair that represents the dimensions of the matrix.

You may ask what use this is. But I will not answer that question right now! First, I should say that gluing together the columns of a matrix is a mathematical operator that even has a name. In matrix differentiation, it is called the $\text{vec}$ operator.

Let us return to the question above: when we represent matrices as vectors, differentiation with respect to matrices becomes differentiation with respect to vectors, which is fundamentally familiar, and we know this from Mathematics 2 in the first year of university. You remember the Jacobian matrix.

In Mathematics 2 in the first year of university, we saw that to differentiate functions with respect to vectors, by definition we proceed as follows:

$$\begin{array}{cc}& f(x)=(f_1(x),f_2(x),\dots f_n(x))\\ & \frac{\partial f}{\partial x}=J=\left[\begin{array}{cccc}\frac{\partial f_1}{\partial x_1}& \frac{\partial f_1}{\partial x_2}& \cdots & \frac{\partial f_1}{\partial x_m}\\ \frac{\partial f_2}{\partial x_1}& \frac{\partial f_2}{\partial x_2}& \cdots & \frac{\partial f_2}{\partial x_m}\\ ⋮& \ddots & ⋮& \\ \frac{\partial f_n}{\partial x_1}& \frac{\partial f_n}{\partial x_2}& \cdots & \frac{\partial f_n}{\partial x_m}\end{array}\right]\end{array}$$

Well, we use this very same method to take derivatives with respect to matrices too. We consider the derivative with respect to a matrix to be the derivative with respect to the $\text{vec}$-ed form of that matrix:

$$\frac{\partial f}{\partial X}=\frac{\partial f}{\partial \text{vec}(X)}$$

More precisely:

$$\frac{\partial F}{\partial X}=\frac{\partial \text{vec}(F)}{\partial \text{vec}(X)}$$

And that’s that. We no longer need tensor algebra, nor those covectors and contravectors! The only thing that descends from tensor algebra down to the level of matrices and that we need is the Kronecker product. For the time being, forget the Kronecker product. First let us enjoy this simplification a bit; there will be time for a deeper look.

Let us differentiate a matrix with respect to itself. To do this, we must turn it into a vector:

$$\begin{array}{cc}& A=\left[\begin{array}{cccc}{A}_{11}& A_{12}& \cdots & A_{1m}\\ A_{21}& A_{22}& \cdots & A_{2m}\\ ⋮& \ddots & ⋮& \\ A_{n1}& A_{n2}& \cdots & A_{nm}\end{array}\right]\\ & \text{vec}(A)=\left[\begin{array}{c}{A}_{11}\\ A_{21}\\ ⋮\\ A_{n1}\\ A_{12}\\ A_{22}\\ ⋮\\ A_{nm}\end{array}\right]\\ & \text{vec}(A{)}_k=A_{\lfloor \frac{k}{n}\rfloor ,k-n\lfloor \frac{k}{n}\rfloor }\end{array}$$

Then it is enough to compute the Jacobian matrix:

$$J_{ij}=\frac{\partial \text{vec}(A{)}_i}{\partial \text{vec}(A{)}_j}={\delta }_{ij}$$

The Jacobian in this case simply becomes the identity matrix with dimensions $nm\times nm$. Well, this was exactly what was expected. Now, a derivative that is somewhat more complicated. Suppose we know the single-variable function $g$. We apply it to all the elements of the matrix $A$, and we call this operation $g$ as well:

$$g(A)=\left[\begin{array}{cccc}g(A_{11})& g(A_{12})& \cdots & g(A_{1m})\\ g(A_{21})& g(A_{22})& \cdots & g(A_{2m})\\ ⋮& \ddots & ⋮& \\ g(A_{n1})& g(A_{n2})& \cdots & g(A_{nm})\end{array}\right]$$

What does the derivative $\frac{\partial g(A)}{\partial A}$ become? With calculations similar to the calculation of $\frac{\partial A}{\partial A}$, we have:

$$J_{ij}=\frac{\partial \text{vec}{(g(A))}_i}{\partial \text{vec}(A{)}_j}={\delta }_{ij}{g}^{\prime }(A_{\lfloor \frac{i}{n}\rfloor ,i-n\lfloor \frac{i}{n}\rfloor })=\text{diag}(\text{vec}(g^{\prime }(A)))$$

This means a matrix whose only nonzero elements are on the main diagonal, and whose values are the values of the derivatives of $g$ at the elements of $A$.

The only thing I should add in this opening post is that differentiation with respect to matrices is also a linear operator. This means that scalar multiplication and addition have the distributive property with respect to this operator:

$$\frac{\partial (\alpha A+\beta B)}{\partial X}=\alpha \frac{\partial \text{vec}(A)}{\partial \text{vec}(X)}+\beta \frac{\partial \text{vec}(B)}{\partial \text{vec}(X)}$$

So far, we have seen that if we temporarily ignore the form of the arrangement of variables inside a matrix, we arrive at a set of well-structured definitions for differentiation with respect to a matrix. To do this, we use the $vec$ operator, which glues together the columns of the input matrix and creates a long vector. For convenience, from now on, wherever I write $\frac{\partial A}{\partial B}$, what I mean is that same $\frac{\partial \text{vec}(A)}{\partial \text{vec}(B)}$. In the next post, I will address more interesting parts of differentiation with respect to matrices.