Gradient Dynamics and Basin Geometry in Overparameterised Models

A puzzling question is why an overparameterised network (which has enough parameters to fit virtually any random labelling of the training data) ends up finding a solution that generalises instead of one of the astronomically many bad solutions (which would overfit).

The answer lies in the geometry of the loss landscape and the trajectory of gradient-based optimisation. In an overparameterised regime, there is typically not a single point that minimises training loss, but rather an entire manifold of global minima (sometimes called a degenerate manifold of solutions). This means that the training equations L( w)=0 (or L( w) = minimal value) are underdetermined, leaving many degrees of freedom in w. For example, one can often permute neurons or rescale weights in certain layers without changing the network’s function and these symmetries create continuous families of equivalent solutions. More generally, if  P >> N (parameters far exceed data samples), the system of equations defining zero training error is highly underconstrained, so a contiguous set (manifold) of solutions in R^P will satisfy it. Crucially, however, not all global minima are equal: they can differ in how they behave on new data. The geometric structure of this solution manifold and the path the optimiser takes determine which particular minimum is selected.

Gradient Flow Bias

Gradient descent (or SGD) is a particular dynamical system that does not pick a random global minimum uniformly; It has biases. One can view the training process as a trajectory w(t) in the weight space, following the negative gradient field −∇ L(w). Despite the nonconvexity of L(w) in standard parameter coordinates, there is evidence that in practice, gradient descent is surprisingly good at navigating towards wide, flat basins rather than narrow ones.

One intuitive explanation, is the volume effect; Wide basins attract more trajectories. Another explanation uses an analogy to energy diffusion: one can think of SGD with small learning rate as a process akin to a particle diffusing on the loss surface while drifting downhill. If there is any stochastic noise (e.g. from mini-batches or explicit noise injection), the process can escape shallow, sharp minima (which occupy tiny volume) but will have difficulty escaping a broad flat minimum once it is found.

In fact, modelling SGD as a stochastic differential equation (Langevin dynamics) shows that in the long-time limit it samples weight space with probability proportional to exp(-L(w)/T) (like a Boltzmann distribution with “temperature” T related to noise). This stationary distribution is heavily influenced by volume in addition to loss; Wide minima correspond to many states of nearly equal loss and thus dominate the probability. The end effect is an implicit regularisation: among all zero-training-error solutions, the training dynamics preferentially diffuses into flat regions of the solution manifold, which correspond to simpler, more stable solutions.

Geodesic connectivity of minima

Another geometric aspect is the concept of geodesic connectivity of minima. Classical thinking portrayed the loss landscape of neural nets as riddled with many isolated “modes” (local minima) separated by high barriers. Yet a surprising phenomenon in deep networks is mode connectivity: many of the minima found by different runs of SGD (or different random initialisations) are not isolated at all, but are connected by continuous paths of low loss. In other words, the apparent multitude of solutions might actually lie on a single, sprawling connected set in weight space.

Empirical studies have shown one can often find a curve (even a simple polygonal chain) in -space along which the training loss remains low, connecting two independently trained network solutions. This suggests the loss landscape, at least in regions that SGD explores, is structured more like one big flat basin with different “funnels” that join at low loss, rather than disjoint multiple minima. Geometrically, this means that the solution manifold is (approximately) path-connected within the low-loss region; any two solutions can be joined by a path that stays near the valley floor.

Why is this important for generalisation? If all these solutions are connected through a flat valley, they likely have similar performance, and crucially, SGD is free to drift within this flat manifold to find a particularly smooth corner. If there were truly separate isolated minima, one might worry about accidentally getting “stuck” in a bad one that overfits. But mode connectivity implies that, in practice, the solutions SGD finds are all part of one good, global basin. Historical concerns about “many minima” in neural nets thus may be misplaced – the minima we actually find form a collective, geometrically simple structure.

Geodesic convexity measures

We can formalise part of this insight with the idea of geodesic convexity in a suitable metric. While L(w) is not convex in the raw parameters w, one can ask if L might be convex along geodesics of some Riemannian metric. For example, consider measuring distance in weight space not by the usual Euclidean norm, but by a metric that reflects changes in the network’s outputs (like the Fisher information metric in information geometry). Two weight configurations w1, w2 might be considered “close” if they represent nearly the same function.

In such a metric, the set of global minima (all of which achieve essentially the same function on training data) might become a single convex (connected, no-barrier) region. Indeed, it has been shown that using natural gradient (which is gradient descent under the Fisher metric) often makes the landscape easier to navigate, as if it were closer to convex in those coordinates.

While a general theory of geodesic convexity for deep networks is still elusive, intuitively the extra degrees of freedom in overparameterised models allow continuous deformations between solutions. One can move from one solution to another by sliding along directions that keep the loss low – essentially traveling within the solution manifold. These directions correspond to perturbations that change weights but leave the network’s function nearly invariant (e.g. swapping symmetrical neurons, or adjusting one layer’s weights while compensating in the next). The existence of such paths further explains why overparameterisation doesn’t ruin generalisation: the optimiser is free to wander and find a flat spot. Rather than being forced into a single narrow optimum, the excess parameters act like a high-dimensional "absorbing cushion" – they accommodate many solutions and connect them, so that the optimiser can ultimately settle in a region that is both optimal and stable.

Summary

In summary, the basin geometry of deep networks is very favourable. Overparameterisation creates a vast connected manifold of near-optimal solutions, and the combination of high-dimensional volume effects and gradient-driven diffusion causes the learning dynamics to end up at flat, robust minima on this manifold. This stands in contrast to the classical picture where more parameters would simply mean more opportunities to overfit. Here, more parameters also mean more pathways to find broad optima and more flexibility for the model to continuously tune itself towards simpler solutions during training. The generalisation ability of DNNs can thus be viewed as a natural outcome of the geometry of gradient flow on these high-dimensional error surfaces, which biases toward functions of lower complexity even without explicit regularisers.