Assessing a Fit

A converged fit is not yet a trustworthy one. Three checks, in increasing strength, tell you whether to believe a result — and they answer different questions. This page is the entry point that ties them together; each step links to where it is documented and driven in full.

1. Is the χ² in its band?

The first, cheapest check is the reduced \(\chi^2\). For a correct model with correctly estimated Gaussian errors, the fit \(\chi^2\) follows the chi-squared distribution with \(\nu = N - N_\mathrm{free}\) degrees of freedom, so a good fit lands \(\chi^2_r\) inside a band around 1 that tightens as \(\nu\) grows. Asymmetry turns the fitted \(\chi^2\) into a two-sided verdict at a confidence level \(R\) (default 0.95):

  • good — \(\chi^2_r\) inside the band;

  • poor (upper tail) — the model is missing physics, or the input errors are underestimated;

  • overdone (lower tail) — the fit reproduces the data better than the errors allow, usually meaning overestimated errors or an over-flexible model.

See Fit statistics for how to read the numbers and what each tail implies. The verdict assumes the \(\chi^2\) was computed against real error estimates; with unit weights or scatter-estimated errors (which force \(\chi^2_r\) toward 1 by construction) it carries no goodness information and is suppressed (see Parameter trending). Inspecting the residual time series is the companion structural check: a good χ² with coherent residual structure still means a missing component.

2. Is the verdict recorded and consistent across the series?

A single number is recorded per fit as a compact, JSON-serialisable summary — success flag, \(\chi^2\), reduced \(\chi^2\), and the fitted parameter values with uncertainties. Both the run-batch and grouped-series recording paths write this same shape, so every fit in a series carries a comparable verdict into the parameter-trending tables (Parameter trending; the shared data model is described under The trending data model). Reading the χ² verdict across a run series is often more informative than any one fit: a χ²ᵣ that drifts upward through a temperature scan flags a model that fits the low-temperature spectra but not the high-temperature ones, and a background parameter that wanders usually signals a changing beam spot rather than sample physics.

3. Are the error bars themselves honest?

The strongest test does not look at χ² at all; it asks whether the reported uncertainties are calibrated. The pull diagnostic re-simulates the fitted run many times at matched statistics, refits each copy, and forms the pull \((\hat\theta - \theta_\mathrm{true})/\sigma_{\hat\theta}\) for every free parameter. For a sound analysis chain the pulls are standard normal: the mean sits at zero (a non-zero mean is a bias) and the width sits at one (a width below one means the errors are too large; above one, too small). This is the definitive answer to “can I quote this uncertainty?” — and the one a low χ² alone cannot give, because a fit can land a perfect χ² while its Hessian errors are mis-scaled.

Before the full pull diagnostic, a quick read on a single parameter’s error is its MINOS asymmetric interval: when the upper and lower excursions differ markedly, the symmetric Hessian error is already suspect — the likelihood is skewed or the parameter is against a bound — and the pull diagnostic is worth running.

Use the three together: the χ² band is a fast triage, the recorded verdict tracks quality across a series, and the pull diagnostic validates the error bars before you quote them.

See also