Variational Distributional Neurons for Measurable Internal Uncertainty in Language Models

Abstract

Large language-model research increasingly relies on scale, which can make mechanism isolation difficult. We study the complementary small-scale regime as a controlled setting for architectural discovery. We examine EVE, a variational distributional neuron that replaces selected deterministic hidden activations with input-conditioned Gaussian latent states and exposes their local mean, variance, and Kullback–Leibler activity during computation. We evaluate EVE on a language-modeling benchmark comprising 1% of the WritingPrompts-Filtered dataset, using five matched random seeds and a reproducible experimental pipeline. Relative to the deterministic baseline, EVE reduces mean test negative log-likelihood by 4.63% (5.359 ± 0.039 versus 5.620 ± 0.022), reduces perplexity by 22.88%, increases next-token accuracy by 11.93%, and improves conditional value-at-risk negative log-likelihood by 3.69%. EVE outperforms the deterministic baseline across all five paired seeds for negative log-likelihood, accuracy, and conditional value-at-risk negative log-likelihood. The paired negative log-likelihood difference is −0.260, with an exploratory 95% interval ranging from −0.330 to −0.191, while the mean calibration error remains comparable. Under covariate non-stationarity, aggregate negative log-likelihood increases from 5.29 to 8.06 as context-token corruption rises from 0 to 0.30, while internal Kullback–Leibler activity increases from 0.274 to 0.362. Validation-selected readouts further reveal a risk-sensitive readout gap. In a representative run, a readout selected for expected calibration error preserves negative log-likelihood while reducing conditional value-at-risk negative log-likelihood from 12.84 to 12.08. A balanced readout increases next-token accuracy to 0.170 while reducing conditional value-at-risk negative log-likelihood to 11.94. These results establish EVE as a small-scale mechanism for measurable internal distributional activity and motivate further parameter-matched and larger-scale evaluations.

Keywords

• variational distributional neurons
• variational inference
• internal uncertainty
• language modeling
• uncertainty quantification
• non-stationarity
• EVE
• posterior activity

Citation

Ruffenach, Yves. “Variational Distributional Neurons for Measurable Internal Uncertainty in Language Models.” Preprint, version 1, July 29, 2026. DOI: 10.5281/zenodo.21669216.