Optimization techniques for deep neural networks

Abstract

Optimization techniques are used to make the neural networks to converge to an optimal solution. Different techniques such as gradient-based approaches are utilized to obtain the parametric solution having objective function estimation to be minimum. The most commonly used method is Gradient Descent (GD) for this purpose, but there are certain drawbacks to this method such as there is no guarantee that it will converge to the global minima of the objective function, the convergence rate depends on the learning rate α (alpha) and wrong selection of which may lead to divergence. The solution depends on the parameter initialization and may differ with different initial values of the parameters, and also, the underlying assumption is that function should be differentiable, which may not always be possible. There are other non-gradient based optimization techniques such as genetic algorithms, graph cut method, particle swarm optimization, simulated annealing. Each of these techniques has their respective drawbacks and advantages. For the current research work, simulated annealing was kept in focus given it guarantees global optimization theoretically (unlike other methods). The gradient-based optimization methods and simulated annealing (SA) algorithm are used on different neural net architectures to check the running time of the algorithm and optimized parameter values indicating convergence of objective function.