目录
二、Introduction of gradient descent cases：
三、Gradient descent formula and its simple understanding:
四、Formula operation precautions:
一、The concept of gradient：
① In a univariate function：
gradient is actually the differentiation of the function, representing the slope of the tangent of the function at a given point
②In multivariate functions：
a gradient is a vector with a direction, and the direction of the gradient indicates the direction in which the function rises the fastest at a given point
二、Introduction of gradient descent cases：
Do you remember the golf course inside the cat and mouse? It looks like this in the animation:
Let's take a look at these two pictures. You can easily see the distant hill, right? We can take it as the most typical example, and the golf course can also be abstracted into a coordinate map:
So in this coordinate, we will correspond the following (x, y) to (w, b) respectively. Then, when J (w, b) is at its maximum, which is the peak in the red area of the graph, we start the gradient descent process.
Firstly, we rotate one circle from the highest point to find the direction with the highest slope. At this point, we can take a small step down. The reason for choosing this direction is actually because it is the steepest direction. If we walk down the same step length, the height of descent will naturally be the highest, and we can also walk faster to the lowest point (local minimum point). At the same time, after each step, we look around and choose. Finally, we can determine this path:
Finally reaching the local minimum point A, is this the only minimum point? Of course not:
It is also possible to reach point B, which is also a local minimum point. At this point, we have introduced the implementation process of gradient descent, and we will further understand its meaning through mathematical formulas.
三、Gradient descent formula and its simple understanding:
We first provide the formula for gradient descent:
In the formula, corresponds to what we call the learning rate, and the equal sign is the same as the assignment symbol in computer program code. J (w, b) can be found in the regression equation blog in the previous section. As for the determination of the learning rate, we will share it with you next time. Here, we will first understand the meaning of the formula:
Firstly, let's simplify the formula and take b equal to 0 as an example. This way, we can better understand its meaning through a two-dimensional Cartesian coordinate system:
In this J (w, b) coordinate graph, which is a quadratic function, since we consider b in the equation to be 0,So we can assume that =
,So, such a partial derivative can be seen as the derivative in the unary case. At this point, it can be seen that when
>0 and the corresponding w value is in the right half, the derivative is positive, that is, its slope is positive. This is equivalent to subtracting a positive number from w, and its w point will move to the left, which is the closest to its minimum value, which is the optimal solution. Similarly, when in the left half of the function, its w will move to the right, which is close to the minimum value, So the step size for each movement is
.
This is a simple understanding of the gradient descent formula.
四、Formula operation precautions:
This is a simple understanding of the gradient descent formula
just like this:
The following is an incorrect order of operations that should be avoided:
This is the understanding of the formula and algorithm implementation for gradient descent. As for the code implementation, we will continue to explain it in future articles.
