# Neural Network — Layered AI Architecture > A computational system of simple units — artificial neurons — organized in layers, each combining weighted inputs and passing the result through a nonlinearity. Individually trivial, collectively a universal function approximator: the substrate on which all of deep learning, and every modern AI model, is built. **Canonical URL:** https://www.andekian.com/ai-lexicon/neural-network **Author / Site:** Stephen Andekian — https://www.andekian.com **Term 52 of 100** · Foundational Architecture **Tags:** Neurons, Layers, Representation, Universal Approximation ## Key Stats - **Unit — neuron:** Weighted sum plus nonlinearity — an operation a spreadsheet could do, repeated billions of times into intelligence. - **Guarantee — universal:** With enough units, networks can approximate any continuous function — the theorem behind their unreasonable generality. - **Key ingredient — nonlinearity:** Without activation functions, any depth collapses to one linear map — the nonlinearity is what makes layers meaningful. ## What Neural Network Actually Is An artificial neuron does almost nothing: multiply each input by a learned weight, sum, add a bias, and pass the result through a simple nonlinear function. The entire edifice of modern AI rests on what happens when this near-trivial unit is replicated — thousands per layer, layers stacked deep, every weight adjustable by training. Complexity is not in the components; it is in the learned configuration of their billions of connections. Layers give the computation its character. Each layer transforms its input into a new representation — and depth composes these transformations into hierarchy. In vision networks the progression is famously legible: edges, then textures, then parts, then objects. In language models, layers move from surface patterns toward syntax, semantics, and task-relevant abstraction. This hierarchical representation learning — features discovered rather than engineered — is the capability that separated neural networks from everything before them. The nonlinearity is the load-bearing detail. Stack any number of purely linear layers and the result collapses mathematically into a single linear map — no depth, no hierarchy, no power. Activation functions (ReLU and its descendants) break the collapse, letting each layer bend the representation space. With nonlinearity and sufficient width, the universal approximation theorem applies: networks can represent essentially any continuous function. Training is the search for the weights that make them represent the right one. Architectures are arrangements of this substrate, specialized by data type: convolutional networks wire in spatial structure for images; recurrent networks once owned sequences; transformers — stacks of attention and feed-forward layers — now dominate nearly everything. The substrate's properties flow downstream into every business conversation about AI: networks are trained rather than programmed, their knowledge is distributed across weights rather than legible in code, and their behavior is verified empirically rather than proven — facts that originate here, in what a neural network fundamentally is. ## How It Works: From simple units to learned functions A neural network computes in layers — each transforming its input representation into a slightly more useful one, composed until raw data becomes an answer. 1. **Input Encoding** — Raw data — pixels, tokens, measurements — becomes numbers the first layer can consume. 2. **Weighted Combination** — Each neuron multiplies inputs by learned weights and sums — the operation where stored knowledge meets incoming data. 3. **Nonlinear Activation** — The sum passes through an activation function — the bend that makes depth mathematically meaningful. 4. **Layer Composition** — Each layer's output feeds the next — representations growing progressively more abstract and task-relevant. 5. **Output Mapping** — The final layer converts the deepest representation into the answer's form — probabilities, values, or tokens. 6. **Training Adjustment** — Backpropagation and gradient descent tune every weight — the search through configuration space for the function you wanted. ## Anatomy: The Components Teams Must Understand - **Artificial Neuron** (The atomic unit): Weighted sum, bias, nonlinearity — deliberately simple, because the power was always going to come from scale and training. - **Weights & Biases** (The learned substance): Every connection's strength — the parameters where all knowledge lives, adjusted by training, opaque to inspection. - **Activation Functions** (The essential bend): ReLU and relatives breaking linearity — without them, a thousand layers compute no more than one. - **Hidden Layers** (Representation factory): The stages between input and output where features are discovered — hierarchy as architecture. - **Width & Depth** (Capacity dimensions): Units per layer and layers per stack — the sizing dials trading expressiveness against compute and trainability. - **Architecture Families** (Specialized arrangements): Convolutional, recurrent, transformer — the same substrate wired for images, sequences, and attention respectively. ## Strategic Implications - **Trained, not programmed** (01 · Paradigm): Neural networks acquire behavior from data rather than instructions — the root fact behind every downstream difference: probabilistic outputs, empirical verification, data as the primary investment. Organizations that internalize this paradigm shift manage AI well; those that manage it like software get surprised. - **Knowledge without legibility** (02 · Opacity): What a network knows is distributed across billions of weights — not readable, not directly editable, not provable by inspection. Assurance is behavioral: testing, monitoring, evaluation. Governance frameworks built for legible code need rebuilding around this fact. - **One substrate, every domain** (03 · Generality): Universal approximation plus representation learning is why the same technology reads scans, writes code, and forecasts demand — and why AI competence transfers across domains. Investments in the substrate's skills and infrastructure amortize over every application it touches. ## Common Misconceptions - **Myth:** “Neural networks are digital brains.” **Reality:** The inspiration was loosely biological; the artifact is matrix algebra. Real neurons are vastly more complex, and the brain shows no evidence of backpropagation. The metaphor explains the name — not the system. - **Myth:** “Bigger networks are better networks.” **Reality:** Capacity must match data and task — oversized networks overfit, underdeliver per dollar, and complicate deployment. Architecture fit and training quality beat raw size routinely; right-sizing is the actual skill. - **Myth:** “Universal approximation means networks can learn anything.” **Reality:** The theorem says a representation exists — not that training will find it, that data carries the signal, or that the result generalizes. The gap between representable and learnable is where all the engineering lives. ## Related Terms - [LLM — Large Language Model](https://www.andekian.com/ai-lexicon/llm) - [Weights & Parameters — Learned Intelligence As Math](https://www.andekian.com/ai-lexicon/weights-and-parameters) - [Transformer Architecture — Modern LLM Foundation](https://www.andekian.com/ai-lexicon/transformer-architecture) - [Sparse Models — Partial Network Activation](https://www.andekian.com/ai-lexicon/sparse-models) - [Gradient Descent — Optimization Algorithm](https://www.andekian.com/ai-lexicon/gradient-descent) - [Backpropagation — Neural Weight Adjustment](https://www.andekian.com/ai-lexicon/backpropagation) - [Deep Learning — Multi-Layer Neural Training](https://www.andekian.com/ai-lexicon/deep-learning) - [Latent Space — Hidden Representation Space](https://www.andekian.com/ai-lexicon/latent-space) ## Explore the Full Lexicon All 100 terms: https://www.andekian.com/ai-lexicon ## Contact Book a conversation or send an inquiry: https://www.andekian.com/#contact LinkedIn: https://www.linkedin.com/in/andekian/