Measuring Internal Probabilistic Activity with Variational Distributional Neurons

Abstract

Variational distributional neurons make internal uncertainty directly measurable by representing each activation as a learned posterior distribution conditioned on the current input. Elemental Variational Expanse (EVE) implements this principle in a language-model unit that exposes local Kullback–Leibler (KL) activity, posterior scale, posterior mean energy, and active-unit fraction during forward computation. A controlled next-token prediction experiment compares EVE with a deterministic model and Monte Carlo (MC) Dropout across five shared random seeds, with a five-member deterministic ensemble serving as an additional predictive reference. Raw EVE achieves a mean Monte Carlo negative log-likelihood of 5.359, compared with 5.620 for the deterministic model and 5.621 for MC Dropout. Mean perplexity is 212.743, versus 275.845 and 276.161, respectively. Mean next-token accuracy is 0.166, versus 0.148 and 0.153, while mean tail-aware conditional value-at-risk negative log-likelihood (CVaR-NLL) is 12.713, versus 13.200 and 13.213. Mean expected calibration error remains comparable across the models, at 0.042 for EVE, 0.043 for the deterministic model, and 0.046 for MC Dropout. Validation-selected readouts provide explicit calibration-oriented, balanced, and tail-oriented operating regimes. As context-token perturbation increases from 0.05 to 0.30, mean internal Kullback–Leibler activity rises from 0.291 to 0.362, posterior scale increases from 0.874 to 0.899, and posterior mean energy rises from 0.052 to 0.075. These coordinated measurements demonstrate that unit-level posterior activity can be quantified alongside predictive behavior during language-model computation. They provide a concrete measurement basis for scaling variational distributional units to larger model architectures.

Keywords

• variational distributional neurons
• language models
• internal probabilistic activity
• uncertainty quantification
• variational inference
• local latent variables
• posterior diagnostics
• calibration
• tail-aware evaluation
• distribution shift

Citation

Ruffenach, Yves. “Measuring Internal Probabilistic Activity with Variational Distributional Neurons.” Preprint, version 1, July 29, 2026. DOI: 10.5281/zenodo.21668942.