# Hyperparameters — Training Configuration Settings > The settings that govern training but aren't learned from data — learning rate, batch size, model depth, regularization strength, epoch count. Parameters are what the model learns; hyperparameters are the conditions under which it learns them, and they can make or break identical architectures. **Canonical URL:** https://www.andekian.com/ai-lexicon/hyperparameters **Author / Site:** Stephen Andekian — https://www.andekian.com **Term 50 of 100** · Training & Optimization **Tags:** Configuration, Tuning, Learning Rate, Search ## Key Stats - **Distinction — set vs learned:** Parameters update via gradient descent; hyperparameters are chosen before training and tuned by experiment. - **Most critical — learning rate:** The consensus highest-impact dial — capable of stalling or destroying a run that every other setting had right. - **Swing — make-or-break:** Identical model and data under different configurations span the range from state-of-the-art to outright failure. ## What Hyperparameters Actually Is Every training run is governed by two kinds of numbers. Parameters — the weights — are learned from data by gradient descent. Hyperparameters are everything the practitioner must set before learning begins: how fast to step (learning rate), how much data per step (batch size), how big the network (depth, width), how hard to resist memorization (regularization, dropout), how long to train (epochs). The model cannot choose these for itself, and they determine whether it learns brilliantly, slowly, or not at all. Their impact is far from cosmetic. The same architecture on the same data can land anywhere from state-of-the-art to broken depending on configuration — with the learning rate as the most notorious single dial. Hyperparameters also interact: optimal learning rates shift with batch sizes, regularization needs vary with model scale, and schedules change everything they touch. Tuning is therefore a search through a coupled, expensive-to-evaluate space — every probe costs a training run. The search itself has become a discipline. Grid search exhausts combinations and scales terribly; random search beats it by spending probes more efficiently; Bayesian optimization models the configuration landscape and spends each expensive trial where information is richest. Mature ML platforms automate the loop, and at frontier scale, labs tune small models and extrapolate configurations upward via scaling relationships — because trial and error at full size would bankrupt the program. Throughout, validation discipline guards the process: configurations chosen against the validation set, verified on untouched test data. The LLM era redistributed the work without retiring it. Teams consuming models through APIs inherit the vendor's training hyperparameters and meet a smaller surface of their own: fine-tuning configurations (learning rate, epochs, adapter rank) where the classic stakes apply in miniature, and inference settings — temperature, sampling parameters, reasoning budgets — which are hyperparameters of behavior rather than learning. The skill transfers intact: change one dial at a time, measure honestly, and respect how much the settings move the outcome. ## How It Works: Configuring the conditions of learning Hyperparameter work is structured search — define the space, explore it efficiently, validate honestly — because theory alone cannot set the dials. 1. **Space Definition** — The tunable dials and their plausible ranges are enumerated — bounding the search before spending compute on it. 2. **Strategy Selection** — Grid, random, or Bayesian search — chosen by budget and dimensionality, since every probe costs a training run. 3. **Trial Execution** — Candidate configurations train — often briefly or at reduced scale, screening cheaply before committing fully. 4. **Validation Scoring** — Each configuration is judged on held-out performance — the comparable number steering the search. 5. **Refinement** — The search concentrates around promising regions — coarse exploration narrowing to fine adjustment. 6. **Final Verification** — The chosen configuration is confirmed on untouched test data — guarding against having overfit the validation set itself. ## Anatomy: The Components Teams Must Understand - **Learning Rate & Schedule** (The master dial): Step size and its evolution across training — warmup, decay, and the single most run-deciding choice in the space. - **Batch Size** (Data per step): Examples per update — trading gradient quality, hardware utilization, and generalization behavior in one number. - **Architecture Dials** (Capacity settings): Depth, width, attention heads — the structural hyperparameters that size the model before data ever flows. - **Regularization Strength** (The memorization brake): Dropout rates, weight decay, label smoothing — the dials balancing fit against generalization. - **Search Automation** (Tuning at scale): Bayesian optimizers and early-stopping schedulers spending trial budgets intelligently — tuning as platform capability. - **Inference Settings** (The deployment-side dials): Temperature, sampling, reasoning budgets — behavior hyperparameters that API consumers own even when training is the vendor's. ## Strategic Implications - **Configuration is capability** (01 · Reality): The gap between a mediocre and excellent result on identical data is often pure configuration — which is why experienced training teams and proven recipes carry the value they do. Budget tuning as part of every training effort; skipping it forfeits performance already paid for. - **Tuning is a portfolio of expensive bets** (02 · Economics): Each configuration trial costs a training run, making search strategy a real money decision — random and Bayesian methods exist to spend fewer runs for better answers. At scale, tune small and extrapolate; trial-and-error at full size is how budgets vanish. - **API consumers still own dials** (03 · Transfer): Fine-tuning configurations and inference settings — temperature, sampling, reasoning budgets — are hyperparameters in the consumer's hands, with real quality consequences. The discipline applies downstream: one change at a time, measured against a held-out suite. ## Common Misconceptions - **Myth:** “Good defaults make tuning unnecessary.” **Reality:** Defaults are starting points calibrated to nobody's task in particular. They prevent disasters; they don't find the performance your specific data and architecture have available. The gap is routinely large. - **Myth:** “Hyperparameter tuning is trial-and-error guesswork.” **Reality:** Modern tuning is structured search — Bayesian optimization, principled schedules, scaling extrapolation — run as an engineering process with budgets and stopping rules. The guesswork era ended with the tooling. - **Myth:** “Once tuned, the configuration is settled.” **Reality:** Optimal settings shift with data changes, scale changes, and architecture revisions — yesterday's recipe quietly underperforms on today's run. Re-tuning checkpoints belong in any evolving training program. ## Related Terms - [Validation Loss — Training Health Indicator](https://www.andekian.com/ai-lexicon/validation-loss) - [Pretraining — Large-Scale Model Learning](https://www.andekian.com/ai-lexicon/pretraining) - [Benchmarking — Standardized AI Evaluation](https://www.andekian.com/ai-lexicon/benchmarking) - [Overfitting — Poor Generalization](https://www.andekian.com/ai-lexicon/overfitting) - [Underfitting — Insufficient Learning](https://www.andekian.com/ai-lexicon/underfitting) - [Gradient Descent — Optimization Algorithm](https://www.andekian.com/ai-lexicon/gradient-descent) - [Epoch — Complete Training Cycle](https://www.andekian.com/ai-lexicon/epoch) - [Loss Function — Measures Prediction Error](https://www.andekian.com/ai-lexicon/loss-function) ## Explore the Full Lexicon All 100 terms: https://www.andekian.com/ai-lexicon ## Contact Book a conversation or send an inquiry: https://www.andekian.com/#contact LinkedIn: https://www.linkedin.com/in/andekian/