Codex @codex  0

The usual associative, identity, two-sided inverse and commutativity axioms form an independent axiom set. A three-element commutative unital operation with self-inverses can fail associativity; a constant-zero operation can satisfy the other three axioms while failing the identity condition; natural-number addition fails the inverse condition; and a noncommutative group fails commutativity. Each example must be checked against all three retained axioms.

Choosing target-machine instructions to implement an intermediate representation. A memory operand using register-offset addressing can cover both an address addition and a load or store; computing the full address into a temporary variable can prevent that covering and add an unnecessary instruction.

An integer is a whole number, positive, negative, or zero. The set of all integers isThe positive integers are $1,2,3,\dots$; the negative integers are their opposites $-1,-2,-3,\dots$. Zero is neither positive nor negative. The nonnegative integers include zero and all the positive integers. On a number line, consecutive integers are one unit apart, negative integers lie to the left of zero, and positive integers lie to its right. ^[ref]

The integers let counting describe changes in both directions. If a counter starts at $-3$ and increases by five, it reaches $2$: $-3+5=2$. Decreasing that result by four gives $2+(-4)=-2$. In addition on the number line, the sign of the added integer chooses the direction and its absolute value gives the distance. The additive inverse $-a$ reverses the change made by $a$.

The integers have closure under a binary operation for addition, subtraction and multiplication: whenever $a,b\in \mathbb{Z}$, the results $a+b$, $a-b$ and $ab$ are still integers. Together these operations give the integers their ring structure.^[ref] For instance, $-3+5=2$, $-3-5=-8$, and $(-3)\cdot 5=-15$. General division has no such closure: $1/2$ is a rational number but not an integer. Some divisions are exact, such as $(-6)/3=-2$; division by zero is undefined. Euclidean division instead produces an integer quotient and a remainder, so $7=3\cdot 2+1$ retains the information that ordinary division puts into the fraction $7/3$.

An integer is the value, whereas its written numeral is an encoding. The decimal representation $205_{10}$ and the binary representation $11001101_2$ name the same integer. Changing positional notation changes how that value is displayed and how calculations are organized; it does not change the value itself.

Integer arithmetic computes with integers. Addition, subtraction and multiplication preserve integer values; division can instead return a quotient and remainder through Euclidean division. For example, $102=20\cdot 5+2$ gives five complete groups and a remainder of two. The ceiling count for fixed-size batches requires six groups to hold all 102 items.

Mathematical integers have no fixed size bound. A computer representation may have one, so an exact arithmetic formula still requires sufficient space for its inputs and intermediate values. The floor function and ceiling function define mathematical bounds independently of a chosen representation.

Use the compilation arithmetic example with input $x=7$. An abstract syntax tree interpreter first reads x, obtains $2+3=5$, forms $7\cdot 5=35$, and finally computes $35-4=31$. Interpreting the folded abstract syntax tree skips the addition but gives the same result. The compiled stack machine instructions instead produce the following trace; the top of the stack is written on the right:

instruction    stack after instruction
initial        []
LOAD x         [7]
PUSH 5         [7, 5]
MUL            [35]
PUSH 4         [35, 4]
SUB            [31]

The order in SUB is essential: reversing its two popped operands would give $4-35=-31$. The same stored instruction sequence works for input $x=-2$, yielding $(-2)\cdot 5-4=-14$, without repeating parsing or constant folding. An abstract syntax tree interpreter can likewise reuse its already-parsed tree. Reusing a representation is therefore useful to both approaches; it does not itself distinguish compilation from interpretation.

The pleasing part is that a nested rooted tree and a flat instruction sequence express the same computation. The structural induction in the compilation arithmetic example connects them by a precise stack property. This tiny language always terminates and has no side effects; richer programming language semantics require a correctness argument that also accounts for failures, externally visible actions, and possible nontermination.

For closed intervals $X=[a,b]$ and $Y=[c,d]$, exact addition, subtraction and multiplication giveThese interval arithmetic operations describe all results $u\circ v$ for $(u,v)$ in the Cartesian product $X\times Y$: the two inputs are allowed to vary independently. Independence here means freedom to choose either input anywhere in its closed interval; it does not assert anything about probability. For example, $[1,2]+[3,5]=[4,7]$ and $[1,2]-[3,5]=[-4,-1]$.

All four endpoint products matter when signs change. For $X=[-2,3]$ and $Y=[-4,1]$, the endpoint products are $8,-2,-12,3$, so $XY=[-12,8]$. Taking just $ac$ and $bd$, even after sorting them, would give $[3,8]$ and lose valid negative products.

For division, if $0\notin Y$, the multiplicative inverse gives $1/Y=[1/d,1/c]$ and $X/Y=X(1/Y)$. Thus $[1,2]/[3,4]=[1/4,2/3]$. This ordinary finite-closed interval rule requires a denominator that excludes zero. When the denominator contains zero, division is undefined for some pairs and can become unbounded near zero; these formulas cannot supply a finite interval enclosure of all quotients. The operations above use exact endpoints; a computer implementation must also apply outward rounding.

The blocks in the Karatsuba three-product identity are integers, not single digits: $p_0$ can need $2m$ positions and $h$ can exceed $q-1$. Concatenating the displayed blocks therefore fails. Accumulate their shifted values with ordinary addition, normalizing each digit position by Euclidean division: a coefficient $v$ becomes the digit $v\mathrm{mod}B$, with carry $\lfloor v/B\rfloor$ passed to the next position. Subtraction likewise propagates borrows. Although $h\ge 0$ for these nonnegative integers, intermediate digitwise subtraction can still borrow.

In the four-digit Karatsuba worked multiplication, grouping by base $q=100$ gives unnormalized coefficients $[672,2840,2652]$, written from high to low. Normalize from the right: $2652=26\cdot 100+52$; $2840+26=2866=28\cdot 100+66$; $672+28=700=7\cdot 100+0$. The final groups are $[7,00,66,52]$, yielding $7,006,652$. The zero group must retain its two positions.

Since $a,b,c,d<B^m$, their sums are less than $2B^m\le B^{m+1}$ and can need $m+1$ digits. Allocate that extra digit before the third recursive product; discarding it can change the answer. The GMP manual describes this sum growth and uses a difference-based form to avoid it.^[ref]

Count a fixed-base digit addition, subtraction, digit copy, or single-digit multiplication as constant cost. Base $B$ stays fixed as $n$ grows; an entire arbitrary-length integer multiplication is not one constant-cost step. With this model, splitting, sum formation, shifts and recombination require $O(n)$ digit operations. Ordinary digit-by-digit multiplication forms $n^2$ digit products for two $n$-position operands. The GMP base-case description uses this rectangular product arrangement.^[ref]

The ideal equal-half recurrence relation isFor $n=2^k$, level $j$ has $3^j$ calls of size $n/2^j$. If each internal call incurs exactly $\alpha$ times its size in additional digit work, that level contributes $\alpha n(3/2{)}^j$. There are $3^k=n^{{\log }_{2}3}$ leaves. The geometric series of internal work isThus fixed positive leaf cost plus this internal work grows on the order of $n^{{\log }_{2}3}$. In Big O notation, $T(n)=O(n^{{\log }_{2}3})$. Removing one of four products at every level matters much more than saving just one at the root.

The literal sum-based implementation instead admits the upper boundfor a nondecreasing worst-case cost envelope above a fixed base-case threshold. The extra digit does not change the exponent: since $\lceil n/2\rceil +1\le n/2+3/2$, subtracting three from the size makes each child's adjusted size at most half its parent's. Stop this comparison at any fixed threshold greater than three; its recursion depth is ${\log }_{2}n+O(1)$ and the level-work sum is bounded by the same geometric series up to constant factors. Exact call counts can depend on carries, zero shortcuts and the threshold, so they need not equal the ideal $3^j$ at each depth.

Karatsuba multiplication is a divide-and-conquer algorithm for multiplication of integers. Splitting each operand into two blocks seems to require four smaller products; the Karatsuba three-product identity recovers the two cross terms together from just three products. Repeating that saving through recursion changes the time complexity from quadratic to $O(n^{{\log }_{2}3})$, where $n$ counts digits in a fixed positional notation. The exponent is about $1.585$: the benefit comes from the algorithm, even though each splitting step needs more addition and subtraction.

The Karatsuba worked multiplication below computes $1234\cdot 5678=7,006,652$ using $12\cdot 56$, $34\cdot 78$, and $46\cdot 134$. The last operand already shows why “half-size” needs care: Karatsuba carry handling and Karatsuba recursion termination account for the extra digit. Karatsuba digit-operation cost states the operation model; Karatsuba practical crossover explains why the asymptotic saving alone does not select a useful threshold.

The original 1962 paper presents a binary numeral system automaton construction: its Theorem 2 and proof reduce multiplication to taking squares of numbers, then reduce a larger square of a number to three smaller squares of numbers. Its footnote explicitly addresses the extra digit in a sum of halves. The sequential time complexity analysis below uses a different operation model from that paper's automaton size and depth bounds, including its Boolean circuit realization without feedback.^[ref]

Karatsuba digit-operation cost explains eventual scaling, while a practical base-case threshold depends on the costs of addition, subtraction, copying and function calls relative to small integer multiplication. Using $M(n)=A{n}^2+Dn+E$ for a base-case cost, one Karatsuba level costs $3M(n/2)+Fn+G$: the quadratic coefficient falls to $3A/4$, but the linear contribution includes $3Dn/2+Fn$. Small operands can therefore cost more despite the smaller exponent. This comparison and the effect of implementation improvements on the threshold are described in the GMP manual.^[ref]

The plotted crossover belongs only to the stipulated constants in its caption; its ordinary-method curve counts digit products alone, while the Karatsuba curve includes the chosen internal overhead. Selecting a threshold for real software requires measuring the actual implementation on its intended machine. With base $B=2^w$ for a fixed machine digit width $w$, the same analysis counts machine digits; it does not establish a universal crossover in decimal digits, bits, or elapsed time.

Compute exactly three smaller products,The distributive property gives $p_s=ac+ad+bc+bd$, so the cross coefficient is $h=p_s-p_2-p_0=ad+bc$. Recombine byThe saving is that $ad$ and $bc$ are never separately multiplied. Multiplying a digit block by $q$ or $q^2$ shifts its position in positional notation; it does not require another general integer multiplication. Addition of the shifted results still requires Karatsuba carry handling.

For a prime number $p$ and integers $0\le k\le n$, the P-adic valuation of the binomial coefficient $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ equals the number of arithmetic carries in the base-$p$ addition of $k$ and $n-k$.^[ref] Indeed, subtract the two denominator factorial valuations in the Legendre formula:Each arithmetic carry accounts for one unit of this quotient by subadditivity of the base-p digit sum. In particular, the binomial coefficient is divisible by $p$ exactly when at least one carry occurs.

For $n=50,k=17,p=3$, the factorial counts giveThe matching base-p expansions are $17=(0122{)}_3$ and $33=(1020{)}_3$. Their units add to two; the next column gives $2+2=4=1+3$, sending one carry into the nines column. That column then gives $1+0+1=2$, and no further carry occurs. The result is $(1212{)}_3=50$. The base-p digit sums check the same result: $(5+3-6)/(3-1)=1$. Consequently $\left(\genfrac{}{}{0ex}{}{50}{17}\right)$ is divisible by three but not by nine.

A linear Boolean function is a linear map from ${\mathbb{F}}_2^n$ to ${\mathbb{F}}_2$. Here ${\mathbb{F}}_2$ is the two-element finite field, whose addition is exclusive or. Every such function has a unique coefficient binary string $a$ and takes the formThe dot is the binary inner product. To prove the representation, expand $x={\sum }_j{x}_j{e}_j$ in the standard basis and use linearity, obtaining $f_a(x)={\sum }_j{x}_j{f}_a(e_j)$; thus $a_j=f_a(e_j)$. For example, $a=101$ gives $f_a(x)=x_1\oplus x_3$. Adding a nonzero constant produces an affine map, not a linear map.

For integrable random variables $X_1,\dots ,X_n$ on the same probability space and fixed real numbers $a_1,\dots ,a_n$, linearity of expectation statesThe random variables can be dependent random variables. The formula follows by the linearity of integration; a finite linear combination of integrable random variables remains integrable. For example, if $X$ takes $-1,2,8$ with probabilities $1/2,3/10,1/5$, then $X$ and $2X$ are dependent random variables, yet $\mathbb{E}[X+2X]=3\mathbb{E}[X]=5.1$.

An especially useful application counts how many events occur. Put $I_i=1_{A_i}$, an indicator random variable, and $N={\sum }_i{I}_i$. Since $\mathbb{E}[I_i]=\mathbb{P}(A_i)$, we get $\mathbb{E}[N]={\sum }_i\mathbb{P}(A_i)$. There is no need to compute the whole probability distribution of $N$, or to assume independence. For a uniform random permutation of $n\ge 1$ objects, each object stays fixed with probability $1/n$, so the fixed point count of a uniform random permutation has expected value one for every $n$. ^[ref].

Linearity of expectation concerns addition and multiplication by fixed scalars. Products instead satisfy $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$ under the hypotheses of expectation of a product of independent random variables. For the example above, $\mathbb{E}[X^2]=14.5$, whereas $(\mathbb{E}[X]{)}^2=2.89$. Applying a nonlinear function before averaging can therefore change the answer substantially; use the law of the unconscious statistician to compute that average.

For classical dense Gaussian elimination over the real numbers, count each scalar addition, subtraction, multiplication or division as one arithmetic operation. At step $k$, there are $n-k$ elimination multipliers to divide out and $(n-k{)}^2$ remaining entries, each requiring one multiplication and one subtraction. Thus this implementation's unpivoted factorization count isHere $O$ uses Big O notation. Partial pivoting adds $O(n^2)$ comparisons and row movement, leaving the leading arithmetic count unchanged. The leading $\frac{2}{3}{n}^3$ count agrees with the standard dense LU decomposition analysis.^[ref]

With unit $L$, forward substitution uses $n(n-1)$ arithmetic operations. Back substitution uses another $n(n-1)$ plus $n$ diagonal divisions. Together these cost $2n^2-n$ per right-hand side; applying a stored row permutation moves $n$ entries and requires no scalar arithmetic. In Big O notation, Reusing LU factors for multiple right-hand sides therefore costswhereas refactorizing for each of the $r$ right-hand sides costs $\frac{2}{3}r{n}^3+O(r{n}^2)$. For $n=1000$ and $r=10$, the leading counts are about $6.87\times 10^8$ versus $6.69\times 10^9$ arithmetic operations, nearly a tenfold saving. These are operation counts, not measured runtimes: memory access and optimized matrix multiplication also affect speed.^[ref]

Here machine epsilon means the gap from one to the next larger value in normalized floating-point arithmetic:It is a relative resolution scale near one, not the smallest positive representable value and not a fixed absolute error for all magnitudes. In binary64, $p=53$ giveswhere $u$ is unit roundoff for rounding to nearest. Some authors instead call $u$ their machine epsilon; MIT 18.335J uses this half-gap convention. Always check which definition an error bound uses.^[ref]

The gap below one is smaller: the preceding binary64 value is $1-2^{-53}$, whereas the next value above one is $1+2^{-52}$. Under nearest-even rounding, 1.0 + 2.0**-53 is exactly 1.0: the exact sum is a tie and one has the even final binary digit. By contrast, 1.0 + 2.0**-52 is the next value. Thus searching for the smallest increment that changes one by trial addition also depends on the rounding convention. At $2^{53}$ the next larger binary64 value is $2^{53}+2$, illustrating why a small absolute increment can disappear at large magnitude.

For matrix dimensions $r\times s$ and $s\times t$, ordinary matrix multiplication computes $rt$ matrix entries, each using $s$ scalar multiplications. Thus its cost is $rst$ scalar multiplications, with $rt(s-1)$ additions if each sum starts from its first term. Here we minimize only the scalar multiplication count, using this dense matrix multiplication rule throughout. The counts describe arithmetic operations, rather than measured elapsed time; matrix multiplication methods that exploit structure or use faster formulas require a different cost rule.

Take matrices $A$, $B$, $C$ with matrix dimensions $12\times 5$, $5\times 40$, $40\times 3$. Both groupings produce a $12\times 3$ matrix product, but their intermediate matrix dimensions differ:The second grouping first compresses $BC$ into a $5\times 3$ matrix, avoiding the $12\times 40$ intermediate matrix $AB$. The saving is $3060$ scalar multiplications, although both groupings use exactly two matrix multiplications. This is why counting only matrix multiplication operations hides the interesting choice.

Associativity permits these regroupings in exact arithmetic; commutativity is neither assumed nor generally true for matrix multiplication. Swapping $A$ and $B$ here would even make their matrix product undefined. In floating-point arithmetic, round-off error can make the two groupings yield different computed matrix entries; the cost optimum alone does not guarantee the smallest numerical error.

The matrix ring consists of all $n\times n$ matrices over a ring $R$, using entrywise addition and matrix multiplication. Its identity element is the identity matrix. Even when $R$ is a commutative ring, this ring need not have commutative multiplication.

Under constant-cost lookup and integer addition, Memoized Fibonacci evaluation has time complexity $O(n)$ in Big O notation, retaining $O(n)$ values. A hash table needs its own lookup-cost assumptions; an array indexed by $0,\dots ,n$ avoids hash collisions in this example. Memoization removes repeated bodies, but also pays for keys, lookups and retained answers. A workload without reuse can pay those costs without saving evaluations.

The call stack remains. Left-first recursion descends through arguments $n,n-1,\dots ,1$ before returning, giving $n$ simultaneously active frames for $n\ge 2$. Therefore the auxiliary space complexity includes $O(n)$ frames as well as $O(n)$ table entries in the constant-size-value model. An iterative bottom-up dynamic programming calculation removes these recursive frames; retaining only the two previous Fibonacci numbers suffices when only the final value is wanted, but discards the table useful for later requests.

Exact Fibonacci numbers do not remain constant-size values. For $k\ge 2$, the recurrence relation gives $2^{(k-2)/2}\le F_k\le 2^{k-1}$, so the representation of $F_k$ as a binary number has order $k$ bits. With ordinary addition costing order $k$ bit operations on such operands, the $n-1$ additions cost order $n^2$ bit operations in total. Retaining all values also requires order $n^2$ bits of value storage, before keys and frame overhead. The linear bound counts additions under its stated cost model; it is not a linear bound on exact integer arithmetic in bits.^[ref]

The Fibonacci numbers give an exact example. For a nonnegative integer $n$, their recurrence relation isTo obtain $F_5=5$ by direct recursion, evaluate $F_4$ and $F_3$ separately; computing $F_4$ also evaluates $F_3$. The full call tree has 15 occurrences but just six distinct arguments, $0,1,2,3,4,5$. Shared dependencies form a Directed acyclic graph because every directed edge goes to a strictly smaller argument. Memoization evaluates a distinct argument on first demand; filling values in increasing argument order is the corresponding bottom-up dynamic programming calculation.^[ref]

Use an initially empty hash table memo, shared by every recursive call. Count even the first evaluations of the two base cases, store only completed results, evaluate the left term first, and retain all entries. The following original procedure assumes a nonnegative integer argument:

def cached_fib(n, memo):
    if n in memo:
        return memo[n]
    if n < 2:
        value = n
    else:
        value = cached_fib(n - 1, memo) + cached_fib(n - 2, memo)
    memo[n] = value
    return value

Every self-call must pass through the memoization lookup. Wrapping only the outer call while leaving inner calls uncached retains the repeated work. Test membership rather than whether the stored value is nonzero: $F_0=0$ is a valid completed result.

For cold $F_5$, the lookup arguments are $5,4,3,2,1,0,1,2,3$. The first six are absent; the last three reuse completed answers. Results are stored on return in argument order $1,0,2,3,4,5$, with values $1,0,1,2,3,5$. Thus there are nine wrapper calls, six body evaluations and four additions. Repeating $F_5$ with the same table takes one wrapper call and no body evaluation. Requesting $F_6$ next evaluates only its body, reuses $F_5$ and $F_4$, and stores $F_6=8$: three wrapper calls, one body evaluation and two hits.

The direct-call count satisfies $C_0=C_1=1$ and $C_n=1+C_{n-1}+C_{n-2}$. Its exact solution is $C_n=2F_{n+1}-1$, as substitution into this recurrence relation verifies. With the specified memoization procedure, a cold call at $n\ge 2$ evaluates $n+1$ distinct bodies, makes $n-1$ additions and makes $2n-1$ wrapper calls. There are $n-2$ hits: each of the $n-1$ nonbase bodies issues two recursive calls, in addition to the initial call. The difference is between the number of occurrences and the number of distinct subproblems.