André-Louis Cholesky Created 2026-10-18 Updated 2026-11-05
André-Louis Cholesky was a mathematician and military surveyor whose computational work grew out of geodesy. During the revision of French triangulation surveys, he developed the method now known as the Cholesky decomposition.[ref]
https://www.polytechnique.edu/bibliotheque/actualites/cholesky-andre-x1895. École polytechnique historical archive, A la section géodésique de l'armée
For a real symmetric positive-definite matrix , the Cholesky decomposition writes with lower triangular matrix and positive diagonal entries. A linear system can then be solved by two triangular substitutions. The link with surveying is natural: reconciling redundant, imperfect measurements by least squares leads to such systems when the design matrix has full column rank.Identity source: www.polytechnique.edu/bibliotheque/actualites/cholesky-andre-x1895
Portrait of André-Louis Cholesky
. Unknown photographer (Commons field: no conegut). Public domain (PD-old as declared on Commons). Source and usage information on the linked Wikimedia file page. Source. Carl Friedrich Gauss Created 2026-10-13 Updated 2026-11-05
Carl Friedrich Gauss (1777–1855) connected number theory, geometry, astronomy and measurement. His work on quadratic reciprocity and the straightedge-and-compass construction of the regular 17-gon made concrete arithmetic structure control a geometric construction. His prediction of the position of Ceres from a short observed orbital arc helped recover the object in 1801, showing why orbit determination and least squares belonged together.[ref]
https://mathshistory.st-andrews.ac.uk/Biographies/Gauss/. University of St Andrews, MacTutor biography, sections on the17-gon, Disquisitiones Arithmeticae and recovery of Ceres
The least squares viewpoint illustrates the bridge: observations rarely satisfy every model equation exactly, so one selects parameters minimizing a sum of squared discrepancies. In differential geometry, Gaussian curvature instead detects an intrinsic property of a surface. These contributions link discrete arithmetic questions, continuous geometry and inference from imperfect observations.
Portrait of Carl Friedrich Gauss
. Gottlieb Biermann (1887), after Christian Albrecht Jensen's 1840 Gauss portrait; Gauß-Gesellschaft Göttingen e.V., photograph A. Wittmann. Public domain (Commons PD-Art / life+100; faithful reproduction). Source and usage information on the linked Wikimedia file page. Source. Centering matrix 2026-11-05
The centering matrix subtracts the arithmetic mean from every component of a real vector. For a positive integer , let have every component equal to one and defineFor , write . Then , so . The output lies in the subspace of vectors whose components sum to zero.
Its orthogonal projection properties follow directly. Set . Since , we have and henceThus is a symmetric idempotent matrix, the orthogonal projector onto . It kills constant vectors and leaves zero-sum vectors unchanged. Its kernel of a linear map is and its rank is ; for it is the zero matrix. The eigenvalues are zero on the constant direction and one on its orthogonal complement.
For an original example, has mean , giving . The centered sum of squares isFor these four observations, the usual sample variance is . Centering removes the mean; it does not itself divide by a standard deviation.
For a data matrix with observations in its rows, centers each column. When , its sample covariance matrix is . Also for any scalar , which proves invariance to adding the same offset to every observation. These identities make the centering matrix useful in least squares and principal component analysis.
Dynamic subgrid-scale model 2026-11-05
A dynamic subgrid-scale model estimates its closure coefficient from the resolved flow as the calculation proceeds. In a common construction, a second, coarser test filter probes interactions among resolved scales, providing information for a subgrid-scale model below the original resolution.[ref]
A dynamic subgrid-scale eddy viscosity model by Massimo Germano and others (1990). Abstract and §2, dynamic coefficient from two filtering levels
Write a bar for the original spatial averaging filter and a hat for the test filter. DefineDirect subtraction gives the Germano identityThe rightmost expression uses only resolved quantities. The identity is exact for these definitions; extrapolating a closure between the filter levels is the modeling step.[ref]
A dynamic subgrid-scale eddy viscosity model by Massimo Germano and others (1990). §2, equations 8–10, resolved stress and algebraic two-filter identity
For instance, suppose a chosen closure writes the deviatoric stresses as and , and assumes can be taken outside the test filtering. Then , where . One way to choose is to minimize . Differentiation giveswhen the denominator is nonzero. This is a least squares fit of a scalar to tensor data. In a one-component illustrative fit, and give .
The coefficient need not remain positive. Depending on the closure, a negative value can permit turbulent backscatter, but a small denominator can also make an estimate unreliable. Averaging, bounds and treatment of weak strain are additional choices. Dynamically fitting a coefficient does not remove these assumptions or make unresolved motion observable.
Householder QR decomposition Created 2026-09-24 Updated 2026-11-05
A Householder QR decomposition introduces zeros below successive diagonal entries using orthogonal reflections. For a nonzero real column segment , set when and otherwise, and defineThen and . The sign choice avoids subtracting nearly equal numbers when constructing . If the segment is zero, skip the reflection.
For with , apply the first reflection to all rows, the second to rows , and so on, extending each by an identity on the earlier rows. This giveswhere is orthogonal and consists of an upper triangular matrix above zero rows. The reflections are stored through their defining vectors, so a full dense need not be formed.[ref]
Fundamentals of Numerical Computation by Tobin A. Driscoll and others. Section 3.4, Householder reflections and successive triangularization
For full column rank, the top block of is invertible and back substitution solves the transformed least squares problem. Orthogonal transformations preserve the Euclidean norm, making this route useful when the normal equations would square a large condition number.
Identifying flow-sensor gain and offset with timed transfers uses a known gas volume delivered over different durations to distinguish multiplicative response from accumulated zero error. In the model , a transfer of volume lasting producesThe observations form a linear regression with design matrix rows . If , gain and offset are identifiable only when the durations are not all equal. Repeating one duration measures the same combination of and more precisely, without separating them.
For two noise-free transfers,Stipulated 3 L transfers lasting 1 and 6 s, measured as 3.020 and 3.120 L, give and . The different durations expose an offset that one transfer could misinterpret as a gain error.
Timing spread and gain-offset estimator covariance quantifies the design. Suppose observations have independent equal volume-error variance , and define . Ordinary least squares givesUnder the stipulated model, wider timing separation improves offset precision. For the two durations above and L, the offset's standard uncertainty is about and the gain's about 0.00406.
The reference syringe volume was treated as known in those formulas. If it is uncertain, repeated sensor tests share that reference uncertainty. Indeed the observations identify only the product when both factors are unknown. Additional repetitions cannot establish an absolute gain without an independently characterized volume. The GUM treats fit uncertainties and common-reference correlations as distinct contributions.[ref]
Guide to the expression of uncertainty in measurement by JCGM. §4.2.5, fitted parameters; §5.2.4, common reference data
The primary spirometry statement's multiple-flow syringe checks test the practical response over different delivery rates.[ref]
Standardization of Spirometry 2019 Update by Brian L. Graham and others. Calibration and Quality Control, 3-L syringe over multiple flows
Actual leaks, flow-dependent nonlinearities and uncertain timing require a richer model than a constant gain and offset.Leading-edge radius estimation from sampled coordinates infers local nose curvature from a finite set of profile measurements. Unlike reading a radius from a drawing, it must account for the coordinate scale, the fitted nose location and the range of points treated as a local model.
Choose coordinates with a smooth nose at the origin and a vertical tangent. A local model iswhere is the true airfoil leading-edge radius. Ignoring the quartic term gives the quotient estimate . Even exact coordinates then givefor the displayed quadratic-plus-quartic model. Moving closer to the nose reduces this model bias, but also makes the horizontal displacement small relative to a fixed coordinate error.
For an original numerical example, set and . At , the exact horizontal displacement is and the quotient gives . If is treated as exact while has an absolute error bounded by , the inferred radius ranges from about to . At , the coordinate-error range narrows to about –, but the exact-coordinate quotient is already biased down to .
Leading-edge radius quotient for a stipulated quartic nose with true radius 3 mm: small offsets amplify coordinate error, while larger offsets retain more quadratic-model bias. Original calculation; not measured airfoil data.
Details. A least squares fit can combine several points. If the origin and are known and errors are assigned to , minimizing with positive weights givesprovided the denominator and fitted coefficient are positive. This fit does not remove an omitted quartic term; the leading-edge radius fitting window controls the tradeoff between local-model error and coordinate noise. An uncertain nose origin or appreciable errors in both coordinates require a correspondingly expanded fitting model.
A practical geometry check uses calibrated profile coordinates, tests several fitting windows and reports the variation alongside measurement uncertainty. Analytic profiles provide an implementation check: the four-digit family has the known relation .[ref]
Applied Computational Aerodynamics by W. H. Mason. Appendix A, equation (A-5), four-digit leading-edge radius
That family relation validates a calculation on a known profile; it does not determine the radius of an unrelated measured nose. Leading-edge radius fitting window 2026-11-05
A leading-edge radius fitting window is the chosen neighborhood of nose coordinates used in leading-edge radius estimation from sampled coordinates. It can be specified by coordinate bounds, arc length or a point-selection rule. The choice controls both how closely a local shape model applies and how strongly coordinate errors affect the inferred airfoil leading-edge radius.
Consider the original local model , with known nose origin, exact coordinates and . A quadratic least squares fit with positive fixed weights givesThe quartic term therefore causes a deterministic coefficient error even with exact observations. For a dense, uniformly weighted window , the ratio of sums tends to . Expanding the window admits more of the omitted shape term.
Shrinking the window has a different cost. For unweighted observations with independent horizontal errors of common variance ,At a fixed number of similarly distributed points, scaling every by one half multiplies this variance by sixteen. Very close points can have tiny horizontal displacements relative to the instrument's coordinate error. The window thus presents a concrete bias-variance tradeoff, not a rule that the smallest neighborhood is always best.
For stipulated mm and , the dense-window model gives fitted radii about 2.9682 mm at mm and 2.7362 mm at mm, before any noise is added. These are calculations on a specified quartic curve, not observed airfoil data. The broader window has less sensitivity to fixed coordinate noise but more quadratic-model bias.
A useful measurement report states the origin estimate, coordinate uncertainty, fitting model and exact window, then examines sensitivity to a few nearby windows. If both coordinates are uncertain or the nose location is fitted, the displayed variance formula no longer covers the complete estimation problem. A known analytic profile supplies an implementation check: the four-digit thickness envelope has the radius relation .[ref] That check cannot remove model error for an unrelated measured nose.
Least squares 2026-10-31
The least squares approach chooses parameters by minimizing the sum of squared residuals. A linear least-squares problem has affine residuals, whereas nonlinear least squares permits nonlinear dependence on the parameters. The Euclidean norm makes the objective ; the factor one-half simplifies its derivatives without changing its minimizers.
Multivariate calibration 2026-11-05
Multivariate calibration relates several measured signals jointly to one or more chemical quantities, using reference samples whose relevant properties are known. Multiple wavelengths, detector channels or other signal coordinates can help distinguish overlapping component responses. The IUPAC multicomponent-calibration report treats both forward response models and inverse prediction, together with uncertainty and model validation.[ref]
Guidelines for calibration in analytical chemistry. Part 2: Multicomponent calibration by Klaus Danzer and others. Sections 2–5; official project description, regression and validation scope
In a forward linear measurement model, writewhere contains analyte concentrations, is a background response, and contains channel sensitivities determined from calibration mixtures. If has full column matrix rank, an unweighted least-squares estimator isThe number of channels alone does not guarantee identification: their response vectors must provide independent information.
For an original example with fixed chosen signal/concentration units, setThe estimated concentrations are . Predicted signals are , so the calibration residuals are . They satisfy : least squares balances the three channels rather than satisfying the first two exactly. This example treats the calibrated sensitivities as known; their uncertainty would add another contribution.
If the channel errors instead have known covariance matrix , generalized least squares replaces the estimate byCorrelated channels do not provide the same information as independent repetitions. Nearly dependent sensitivity columns also amplify noise, connecting wavelength selection with the condition number of the inverse problem.
An inverse model can instead fit concentrations directly from measured signal vectors. Methods such as principal component regression reduce the signal representation, but the retained variation must still predict the intended chemical quantity. Cross-validation and separately prepared mixtures test prediction; a high calibration fit alone cannot establish transfer to another sample matrix. Preparing standards in which component concentrations vary independently helps expose confounding between analytes, rather than teaching the model only one fixed mixture ratio.
Numerical linear algebra Created 2026-09-24 Updated 2026-11-05
Numerical linear algebra develops algorithms for linear systems, least squares, eigenvalue problems and other matrix computations, with attention to roundoff error, cost and storage. A mathematically equivalent formula can behave very differently in floating-point arithmetic: solving through an LU decomposition avoids forming an explicit matrix inverse, while a QR decomposition gives an orthogonal route to least squares.
The central distinction is between accuracy of the computed answer and accuracy of the problem it solves. For , an approximate vector has linear-system residual . A small linear-system residual need not imply a small forward error when the condition number is large. For the exact examplethe solution is , whereas has Euclidean norm residual but Euclidean norm error . Here the condition number in the spectral norm is . The lost component barely changes the equations because its coefficient is tiny.
Direct linear solvers such as Gaussian elimination factor a matrix; iterative methods such as conjugate gradient successively improve a vector. For a large sparse matrix, fill-in and matrix preconditioning can matter as much as arithmetic count. The task is to preserve the structure that makes a computation accurate and affordable.
Parameter estimation 2026-11-05
Parameter estimation uses observations to infer the unknown parameters of a mathematical model. The result depends on both the observations and the model: one must specify which quantities are unknown, how observations relate to them, and how discrepancies are assessed. Least squares and maximum likelihood estimation provide two common criteria.[ref]
NIST and SEMATECH e-Handbook of Statistical Methods. §4.1.4.1, Linear Least Squares Regression, unknown parameters and squared deviations
For an explicit example, suppose , where the same unknown level applies to all observations. MinimizinggivesSince , this is the unique minimum for . Observations give the sample mean , with residuals . Independent normal distributions with a common variance also make this estimate the maximum likelihood estimation solution for the level.
An estimate is not the same as a known exact value. Under the independent equal-variance model, its standard error is estimated by , where is the sample variance. For the three illustrative observations, and the estimated standard error is . Correlated errors or a common measurement offset require different accounting; collecting more identically biased readings does not remove the offset.
Identifiability is a separate prerequisite. If a model depends only on , data of that form can estimate their sum but cannot uniquely separate from . Pairs and give the same sum. More repeated observations of that same relationship reduce uncertainty about the sum without identifying both parameters. State what the observations can determine before reporting numerical estimates.
Minimizing the squared residual norm gives the normal equationsFull column rank makes invertible. With columns corresponding to , the design matrix isSymmetry gives zero cross products between the odd column and either even column, and gives the remaining orthogonality. Their squared norms are seven, 28 and 84, so . ThereforeSince the error covariance is , least squares has covariance . ThusThe estimators have zero covariances. independence would require an additional distributional assumption such as Gaussian errors; the stated mean and variance assumptions alone give only the covariance calculation.
Phase-shifting interferometry 2026-11-05
Phase-shifting interferometry measures an optical phase difference by recording light intensities at controlled relative wave phase shifts. For two sufficiently coherent fields at one detector point, a common model iswhere is a common background, a modulation amplitude and a known applied step. Several measurements separate the unknown wave phase from light intensity background and contrast; a single bright or dark frame generally cannot do so.
A concrete primary implementation was reported in 1974. A piezoelectrically varied reference arm and a 1024-element photodiode array supplied sampled light intensities to a minicomputer, which recovered wave phase by synchronous Fourier-series detection. The paper describes optical-surface and lens testing, repeated sampling and compensation of instrumental wavefront errors.[ref]
Digital Wavefront Measuring Interferometer for Testing Optical Surfaces and Lenses by J. H. Bruning and others. Abstract and p. 2693, apparatus and phase-detection method
This is an experimental implementation of controlled wave phase modulation; it is not a claim that every such system uses the same number of steps.Writing the model asturns wave phase recovery into a linear fit followed by . The step choices must make the three columns independent. Repeating only the same wave phase cannot identify all three unknowns. With more steps, least squares can fit the coefficients while exposing model residuals.
A reflecting reference mirror moved axially by changes its round-trip optical path by in the simple normal-incidence geometry, giving . A quarter-cycle wave phase step thus corresponds to ; at 632.8 nm this is 79.1 nm. These are geometric model values, not a calibration result for the historical apparatus.
Phase-step calibration error, light intensity drift, motion between frames and detector nonlinearity can all break the common-intensity model. The retrieved wave phase remains modulo , so phase unwrapping requires additional spatial or temporal information. The useful observable is the relative wave phase under controlled acquisition conditions, not an absolute optical wave phase inferred from brightness alone.
Raman baseline-induced center shift 2026-11-05
A Raman baseline-induced center shift is a displacement of an estimated band center caused by the spectral background used or omitted in the analysis. It is one mechanism of Raman peak-model bias. A background slope can move the local maximum of a band-plus-background signal even if the underlying band has not shifted. A fitted center can also move when an incomplete background model transfers part of that slope into the fitted band position.
For an original local model, write for Raman shift relative to the true center, and consider a Gaussian function band plus a small residual slope:Its stationary-point equation near the band center is . When the displacement is small compared with , the local maximum is displaced byFor intensity units, and intensity units per , this is approximately . A positive residual slope moves this local maximum toward increasing Raman shift. This local spectral-window model does not assert a physically unbounded linear background.
A least squares fitted center answers another question. If is the derivative of a chosen band model with respect to its center, omitted residual background produces the first-order fitted displacementfor fixed other parameters and positive weights . An omitted background component matters when it projects onto the center-shift response. For symmetric samples and weights, a constant residual is orthogonal to the odd derivative of a symmetric band, whereas a residual slope is not. Asymmetric sampling or weights can remove even that constant-background cancellation.
The three-channel fit in Raman peak-model bias gives approximately for the same slope and band parameters, because estimating a center from those samples is not the same operation as finding the continuous local maximum. At sample positions , the center derivative is proportional to ; allowing both a center change and an unknown linear background therefore introduces a first-order identifiability problem in that restricted model. Additional spectral samples or independent background information can help separate their responses.
Examining residuals and changing physically justified background models are useful tests before interpreting a Raman peak-position map as a map of material changes. Manufacturer guidance distinguishes spectral intensity, background contamination and fitted curve parameters.[ref]
What Raman images can reveal by Renishaw. “How can I analyse Raman images?”, intensity/background ambiguity and curve-fit parameters
A small conditional Raman peak-fit uncertainty alone does not establish that the assumed background is adequate.[ref] Raman peak-fit uncertainty 2026-11-05
Raman peak-fit uncertainty characterizes the uncertainty of a spectral-band position estimated by fitting a model to a Raman spectrum. It accompanies a Raman peak-position map: a colour difference between pixels need not establish a physical shift if the estimated positions are insufficiently precise. In Raman mapping, the instrument collects spectra at successive sample positions, and fitting extracts band parameters such as position and width.[ref]
What Raman images can reveal by Renishaw. “How do we collect Raman images?”; “How can I analyse Raman images?”, curve-fit parameters
For an isolated band, a least squares fit might estimate the center, height, width and background of a Gaussian function. The uncertainty assigned to its center is conditional on the chosen model and description of spectral noise. Repeating the complete acquisition and fit at one unchanged location provides another assessment: if independent center estimates have standard deviation , the estimated standard uncertainty of their sample mean is . This reduction applies to independent variation; it does not eliminate an offset shared by every acquisition.[ref]
Guide to the expression of uncertainty in measurement by JCGM. §4.2.1–4.2.3, independent repeated observations and standard uncertainty of their mean
For an original numerical example, model two fitted positions aswhere represents a shared uncertain additive correction from measurement calibration, while are independent fitting errors, also independent of . Suppose their standard uncertainties are and . Each position then has combined standard uncertainty . Their difference cancels the same , givingHere denotes standard uncertainty, so the difference has uncertainty about . The covariance term is essential: treating the two positions as entirely independent would instead give about . A common additive correction cancels only insofar as it really is shared; instrument drift between measurements or a scale error need not cancel.[ref]
Guide to the expression of uncertainty in measurement by JCGM. §5.2, equations13–16 and §5.2.4, correlated inputs and common measurement references
For a hypothetical stress model with an exactly known illustrative , this difference uncertainty contributes to the standard uncertainty of stress. Uncertainty in would require its own contribution. Raman peak-model bias, overlapping bands, an inadequate background and laser-induced sample changes can invalidate an otherwise small fit error. Retaining spectra and repeating fits with physically defensible models therefore matters alongside quoting a numerical uncertainty.
Raman peak-model bias 2026-11-05
Raman peak-model bias is a systematic displacement of an estimated Raman shift or another fitted band parameter caused by an inadequate spectral model. The discrepancy can persist in repeated low-noise measurements. It differs from the random scatter contributing to Raman peak-fit uncertainty. Manufacturer documentation explains that fitted band parameters depend on theoretical curves and warns that a broad spectral background can contaminate simpler intensity measurements.[ref]
What Raman images can reveal by Renishaw. “How can I analyse Raman images?”, curve-fit parameters and background contribution
An original three-channel example isolates a Raman baseline-induced center shift. Write for displacement from the true band center and let the true sampled signal be a Gaussian function plus a small residual baseline slope:Suppose an equal-weight least squares fit omits that slope and estimates only the center , keeping and fixed:Near , its derivative with respect to the center is , so linearizing gives . Minimizing the resulting squared residuals givesThe odd slope in the unmodeled background therefore projects onto the odd center-shift response of the symmetric band.
For intensity units, and intensity units per , the first-order center bias is about . This is a deliberately restricted toy fit, not a universal bias formula: fitting the background, width and height too, or using other sample positions and statistical weights, changes the result. The calculation nevertheless demonstrates why reducing random noise alone cannot remove a wrong-model displacement.
In Raman mapping, a spatially varying background or unresolved neighboring band can consequently create an apparent Raman peak-position map contrast. Comparing physically defensible background/line-shape models and examining their residuals tests that interpretation. A good residual plot is useful evidence but does not prove uniqueness of a spectral decomposition. The GUM requires significant corrections and inadequately represented influences to be included in the measurement model; its discussion of fit-parameter uncertainties does not make an incomplete model adequate.[ref]
Guide to the expression of uncertainty in measurement by JCGM. §4.1.2, incomplete measurement models; §4.2.5, least-squares fit parameters
Reprojection error 2026-11-05
A reprojection error compares an observed image point with the position predicted by projecting a reconstructed scene point through a camera model. If is an observed position and its prediction, the residual is . One possible aggregate is the per-point root mean square Euclidean norm:Coordinates in pixels give an error in pixels. Least squares instead minimizes the associated sum of squares; normalization conventions should be stated when reporting a summary.[ref]
Camera Calibration in the OpenCV-Python tutorials. Re-projection Error, projecting known points and comparing image positions
For an observation and prediction , the Euclidean norm of the residual is 5 pixels. Camera calibration can adjust model parameters to reduce many such residuals, while stereoscopic triangulation can adjust a reconstructed point with the camera parameters fixed. Low reprojection error tests agreement with observations under that model; incorrect stereo correspondence or an incorrect camera baseline can still produce wrong scene geometry.
Ternary tie-line consistency residual 2026-11-05
A ternary tie-line consistency residual quantifies how far an overall three-component composition is from the mass balance predicted by specified endpoints of a ternary liquid-liquid tie line. It is a geometric consistency check on a proposed two-phase decomposition. A thermodynamic tie line connects coexisting equilibrium compositions; the calculation below checks the lever rule for those endpoints, rather than independently proving that they are equilibrium phases.[ref]
Tie-line in IUPAC Gold Book. Entry 15367: tie line connects compositions of independent coexisting equilibrium phases
Use mole fraction vectors for the endpoint liquids and for the overall composition, all normalized to sum to one. Put . For a proposed mole-based phase fraction , defineExact mass balance requires every component of to vanish. Closure gives automatically, so this residual has only two independent components. A zero sum alone does not imply that the component balances hold.
When the phase fraction is unknown and the endpoints are distinct, an unweighted least squares tie-line phase-fraction fit minimizes the squared Euclidean norm of . Differentiating givesThis is an orthogonal projection onto the line through the endpoints. A physically admissible phase fraction also lies in ; if the unconstrained fit lies outside that interval, the closest admissible composition is an endpoint of the line segment. Coincident endpoints do not determine a phase fraction through this formula.
For the hypothetical endpoints and , the composition gives and zero residual. Changing the overall composition to preserves closure but givesNo adjustment of a single phase fraction can eliminate that discrepancy under the stipulated exact endpoints. For experimental use, its significance depends on measurement uncertainty in both the overall composition and endpoint compositions. Unweighted distance is a chosen metric, not an automatic statistical rejection threshold; correlated composition errors require their covariance matrix on the independent composition coordinates. Even a tiny residual establishes only compatibility with this mass balance, not equality of chemical potentials or stability of the proposed liquid-liquid equilibrium.
Timing spread and gain-offset estimator covariance describes how delivery durations affect both precision and correlation when a known volume calibrates an integrated volumetric flow rate sensor. In the model , assume and durations are known and independent volume errors have common variance . Define .
The least squares slope estimates offset , while the intercept divided by estimates gain . Their covariance isThe negative covariance reflects a tradeoff: a larger fitted offset can be compensated by a smaller gain while preserving responses near the middle of the observed durations.
For an original design with L, durations 1 and 6 s, and L, the standard uncertainties are approximately 0.00283 L/s for offset and 0.00406 for gain. Their correlation is about . Moving the second duration to 2 s, with everything else stipulated unchanged, makes the offset uncertainty five times larger. Wider duration spread supplies more leverage for separating slope from intercept within this model.
Prediction shows why the covariance matters. For a new duration , the fitted mean response is , withAt the midpoint 3.5 s in the example, its standard uncertainty is L. Incorrectly dropping the gain-offset covariance would give about 0.0157 L. This is uncertainty of the fitted mean; an independent future observation adds its own error variance.
Centering time at makes the fitted midpoint response and slope uncorrelated under the stated equal-variance model. Transforming back to physical gain restores the covariance, so a change of coordinates does not remove shared uncertainty. The GUM explicitly retains such covariance terms in propagation.[ref]
Guide to the expression of uncertainty in measurement by JCGM. §5.2.2, equation 13, correlated-input propagation
Reference-volume uncertainty, uncertain timing and nonlinear volumetric flow rate response require additional terms beyond this original regression calculation.

