Multiple Dimension Gradient Descent

The extension to multi-dimensional $\Theta$ from 1D Gradient Descent is fairly straightforward.

Let us assume our parameters $\Theta \in {\mathbb{R}}^m$, so that our objective function $f:{\mathbb{R}}^m\to \mathbb{R}$. The gradient of $J$ with respect to $\Theta$ is:

$${\nabla }_{\Theta }f=\left[\begin{array}{c}\frac{\partial f}{\partial {\Theta }_1}\\ ⋮\\ \frac{\partial f}{\partial {\Theta }_m}\end{array}\right]$$

The algorithm remains the same as the 1D case, except that the update step (line 5 in 1D Gradient Descent) becomes:

$${\Theta }^{(t)}={\Theta }^{(t-1)}-\eta {\nabla }_{\Theta }f({\Theta }^{(t-1)})$$

Remember that the gradient points in the direction of the greatest rate of increase. Thus, we take the negative of it to descend.