15 Ways to Apply Fractional Calculus in Gradient Descent Optimization Methods

Many have attempted using fractional calculus (FC) in optimization and machine learning in the past several years. This paper offers a holistic view on how to introduce fractional calculus in gradient descent (GD) optimization methods. We identified four different entry points to add FC ideas: 1) The updating law that could be a fractional order integrator or, in general, a fractional order system; 2) The GD gain coefficient (also known as the learning coefficient) that could be a time-varying signal as an output from a fractional order system driven by an impulse; 3) The gradient that, of course, could be a fractional order gradient based on fractional order partial derivatives; 4) The noise (additive or multiplicative) introduced artificially to the gradient that could also be fractional order stochastic processes. Therefore, in combinations, we have a total of 15 ways to apply fractional calculus in gradient descent optimization methods. We surveyed the existing sample research attempts and categorized them in one of the 15 ways in a big table. It is interesting to show what are those empty spots left on the table hinting rich future research opportunities. Copyright (c) 2025 The Authors. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)