If the Hessian matrix $\b{F}(\b{x}^{(k)})$ is not positive definite, then the search direction $\b{d}^{(k)} = - \inv{ \b{F}(\b{x}^{(k)})}\b{g}^{(k)}$ may not point in a descent direction. A simple technique to ensure that the search direction is a descent direction is to introduce the Levenberg-Marquardt modification to Newton’s algorithm:
$$\b{x}^{(k+1)} = \b{x}^{(k)} - \inv{(\b{F}(\b{x}^{(k)}) + \mu_k \b{I})} \b{g}^{(k)} $$
where $\mu_k \geq 0 $.
The idea is as follows. Consider a symmetric matrix $\b{F} $, which may not be positive definite. Let $\lambda_1, …, \lambda_n$ be the eigenvalues of $\b{F} $ with corresponding eigenvectors $\b{v}_1, \b{v}_2, …, \b{v}_{ n } $. $\lambda_1$, where $\lambda_2, …, \lambda_{ n } \in \R$.
Next, consider the matrix $\b{G} = \b{F} + \mu \b{I} $, where $\mu \geq 0 $. Note that the eigen values of $G $ are $\lambda_1 + \mu, …, \lambda_2 + \mu $, because
$$\begin{align*} \b{G} \b{v}_i &= (\b{F} + \mu \b{I}) \b{v}_i \br &= \b{F}\b{v}_i + \mu \b{I} \b{v} _i \br &= \lambda \b{v} _i + \mu \b{v}_i \br &= (\lambda_i + \mu) \b{v}_i \end{align*}$$
Therefore, if $\mu$ is sufficiently large, then all the eigenvalues of $\b{G} $ are positive, are $\b{G}$ is positive definite. Accordingly, if the parameter $\mu_k $ in the Levenberg-Marquardt modification of Newton’s algorithm is sufficiently large, then the search direction $\b{d}^{(k)} = - \inv{(\b{F}(\b{x}^{(k)}))} \b{g}^{(k)} $ always points in adescent direction. In this case if we further introduce a step size $\alpha_k $ as in Newton’s method,
$$\b{x}^{(k+1)} = \b{x}^{(k)} - \alpha_k \inv{(\b{F}(\b{x}^{(k)}) + \mu_k \b{I})} \b{g}^{(k)} $$
then we are guaranteed that the descent property holds.
The Levenberg-Marquardt modification of Newton’s algorithm, by letting $\mu_k \to 0 $, can be made to approach the behavior of the pure Newton’s Method.
On the other hand, by letting $\mu_k \to \infty $, the algorithm approaches a pure gradient method with small step size.
In practice, we may start with a small value of $\mu_k $, and then slowly increase it until we find that the iteration is descent, that is, $f(\b{x}^{(k+1)}) < f(\b{x}^{(k)}) $.