I defended my PhD: convergence analysis of training deep linear neural networks
The learning rate depends on constants existing theory can't compute before training, so they get chosen by trial and error. I closed part of that gap, and I'm now applying the same rigor to production AI.
On June 15 I defended my PhD at LMU Munich: Convergence Analysis of Training Deep Linear Neural Networks.
Why it matters
Every AI model learns by adjusting itself over millions of small steps. Convergence is the guarantee this process reaches the right answer. For real AI systems, that guarantee has been incomplete.
The learning rate, meaning how big each step should be, depends on constants existing theory can’t compute before training. They’re chosen by trial and error. When training fails, engineers spend weeks debugging by intuition.
I studied this on linear neural networks, whose loss stays mathematically hard to optimize and which share key training behavior with the nonlinear networks used in practice.
I dedicated my PhD to closing part of that gap: the distance between how we train neural networks and why it works just got smaller.
Now I’m applying that same rigor to production AI. I built an agentic supply-chain copilot that puts deterministic guarantees around LLM decisions.
How it works
Prior results for gradient descent (GD) and stochastic gradient descent (SGD) relied on strong conditions such as the Lipschitz gradient property, which doesn’t hold unless the iterates are already bounded, the very thing those results assume. Violating some was shown to cause divergence.
I proved convergence under milder conditions. For GD: to a global minimum for almost all initializations. Stepsizes do not decay exponentially with depth, and every constant is computable from the network and data.
For SGD: one of the first iterate bounds for a concrete neural network loss, with almost sure convergence established.
Do the findings extend? I trained nonlinear networks (tanh, at depths 2, 3, and 5) under the same conditions. Same convergence pattern. Same depth ordering. The linear findings extended empirically.
One surprise: some stepsizes violating our conditions still converged, while others failed completely. That boundary is the next open question.