Trainable Models: Last Station

Written on August 31, 2017

NOTE: This post is automatically translated from Farsi using GPT 5.5

Now that in the previous two parts (first and second) we have gone from the starting point to the practical use of the simplest learnable function, it is better to close the file on this subject at very high speed; because first, long series bore people, and second, even if I wrote a book here about learnable functions, the discussions would not end. Of course, let me admit right now that I will come back to this very subject again, and many times — in the future, of course — because one of my interests is artificial intelligence, and these models are its foundation.

Perhaps it would not be bad, before starting the discussion, to point out why I called the lines separating the tribes in the previous post support lines. Well, in fact I did not choose this name; before me, other friends translated Support Vector Machine as “support vector machine,” which is one type of learnable model and works based on the very thing we saw in some detail in the previous post. I do not know a better name for these lines either.

Real Learnable Models

But to the main point. The functions that are chosen for use in practice are usually much, much more complex than a straight line. Functions whose number of parameters is on the order of ten million. For example, in deep neural networks used for machine translation, there are more than 8 million parameters (see this paper.)

This enormous difference in the number of parameters, although it does not change the main concepts, drastically changes the technical implementation details. For instance, in cases where we have simple functions with a small number of parameters, saddle points and local minima in the function’s domain are far, far fewer, and for this reason optimization algorithms reach the optimal point with a higher probability. At the same time, optimization methods that operate based on second-order derivatives and have a high convergence speed require much more memory for functions with a very large number of parameters. If the terms I used are not very familiar to you, do not worry at all; the gist of what I said is that optimization and finding optimal parameters become much more difficult.

But our world has advanced a great deal compared with the first time the primal-dual method was used in practice. Before this, besides the need for a minimal amount of knowledge to compute the derivative and second-order derivative of complex functions, there was a need for processing resources beyond the public’s reach for training real functions to become practically possible. Now, however, with a suitable GPU priced at around 1.5 million tomans, you can use new software libraries that themselves do all the work related to derivative calculation and optimization for you, and write exciting programs.

If you are interested in topics related to artificial intelligence, I strongly recommend becoming familiar with PyTorch. This rich library, written for the Python language, can greatly speed up research in artificial intelligence. When you define a model using this library, both differentiation and optimization can be done — simply — through the tools it provides you. In addition, running the training and testing of the models you design will happen by executing a handful of commands on the GPU. Something that previously required more knowledge and much, much more work.

Another noteworthy thing is the good, practical examples that the development team has written using this very library and made publicly available. These examples can be obtained from GitHub. Before trying to run the examples, do not forget to install PyTorch itself! The commands related to installing PyTorch can be seen on the front page of the site in the “Run this command:” section.

To give you more motivation to check out the examples, I will list here the titles among them that I found more appealing:

1. Super Resolution (a kind of image quality enhancement)
2. Language Model (a tool for generating text in a particular style and scoring the quality of sentences)
3. MNIST (recognition of handwritten English digits)

Another cute example is image style transfer:

whose source code and explanations you can find here.

Why the Learnable Models Series Came About

Over these past few years that I have worked and done research in fields related to artificial intelligence, it had often happened that I did not like my friends’ and colleagues’ angle of view toward machine learning. What I mean is that when problems fell outside their common form — for example, training was raised in a new way, or a new function was proposed, or even the connection between manifold learning and artificial neural networks was pointed out — that angle of view would get in the way of deep understanding (at least I thought so!)

For this reason, I tried, apart from technical details, to write something that shows the way machine learning works with greater clarity. That machine learning is the same as finding the minimum point of a mathematical function, a simple type of which we also see in high-school mathematics. That if we show neural networks with beautiful graphs reminiscent of the connections of neurons inside the brain, we should not forget that the mathematical relations related to the connection between two of its layers are simply $y=\sigma (Wx+b)$.