mHC: Manifold-Constrained Hyper-Connections

Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

A collection of data examples used for training, validating, or testing machine learning models.

The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.

A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively.

Skip connections that allow gradients to flow directly through a network, enabling training of very deep networks (ResNet).

A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

Learning useful features or representations of data automatically, rather than hand-crafting them.

A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.