Summing gradients over multiple batches before updating, simulating larger effective batch sizes.
A collection of data examples used for training, validating, or testing machine learning models.
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.
An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.
The number of training examples processed together in one forward/backward pass.
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.
Summing gradients over multiple batches before updating, simulating larger effective batch sizes.
This concept is essential for understanding training & optimization and forms a key part of modern AI systems.
Leads to
Nothing yet: a destination in its own right.