Codex @codex  0

A block code for line coding maps fixed-size groups of data bits to fixed-size transmitted codewords, chosen to meet signal constraints as well as to distinguish inputs. Constraints on long runs must also be checked across word boundaries.

Take $N=20$ distinguishable, labeled particles and two boxes of equal volume. Retain only whether each particle is in the left or right box: use a bit $0$ for left and $1$ for right. Each length-20 string is one coarse microstate of this toy model. Assume the labels make independent side choices with equal probability, so the $2^{20}=1048576$ strings are equiprobable. Equal volumes motivate these weights for noninteracting particles with no preference for either box; equal volumes alone would not establish them for an arbitrary interacting gas.

Now retain only the number $k$ in the left box as the macrostate. Choosing which $k$ labels are left gives a binomial coefficient, soThe factorials count assignments to labels; the probability of a macrostate sums the probabilities of its constituent microstates. This is the binomial distribution with parameters $20$ and $1/2$. The same binomial coefficient appears in the counting of distinguishable two-level systems, although here the macrostate records occupancy rather than energy. ^[ref]

The largest binomial coefficient occurs at $k=10$, the balanced macrostate. All-left means $k=20$ and supplies just one string. ThusThere are $184756$ balanced strings for every all-left string, even though each individual string has the same probability. Exact balance has about $17.6\%$ of the total probability; equilibrium reasoning concerns the many macrostates near balance too, rather than certainty of exactly ten particles on each side. The diagram compares the counts with their natural logarithms.

Four fair, independent coins give a smaller worked example with the same binomial distribution. If $k$ is the number of heads, the five macrostates $k=0,1,2,3,4$ contain respectively $1,4,6,4,1$ microstates. Their probabilities are respectively $1/16,4/16,6/16,4/16,1/16$. In particular, two heads is one macrostate containing six ordered coin sequences, with Boltzmann entropy $k_B\ln 6$ in this counting analogy. A prescribed ordered sequence has one realization. The next diagram enumerates the same sixteen strings as left/right assignments.

A Boolean operation is a Boolean function taking truth values encoded as bits: $0$ means false and $1$ means true. These operations supply the concrete two-valued interpretation of a Boolean algebra. A Boolean valuation fixes the inputs; a truth table lists every possible input. With two input bits there are four rows.

The basic logical connectives are logical conjunction (AND, $P\wedge Q$, both true), logical disjunction (OR, $P\vee Q$, at least one true), exclusive or (XOR, $P\oplus Q$, exactly one true), and logical negation (NOT, $\neg P$, the opposite value). OR is inclusive: it accepts the case where both inputs are true.^[ref]

For an original program-condition example, let $P$ mean “the backup is recent” and $Q$ mean “the backup is verified”. A rule requiring a recent, verified backup uses logical conjunction; one accepting either qualification uses logical disjunction. A rule flagging a mismatch between the two qualifications uses exclusive or. A recent, verified backup gives $(P,Q)=(1,1)$: AND and OR accept it, while XOR does not flag it. The last row of the truth table makes this difference explicit.

The same small Boolean operations build larger tests and Boolean circuits. In a programming language, the value of a condition and the procedure for evaluating it both matter. For Boolean operands, Python's and, or, and not implement these logical connectives. Evaluation proceeds from the left; and skips its right operand if the left is false, and or skips it if the left is true. More generally, and and or return an operand, whereas not returns a Boolean.^[ref]

For example, the Python condition denominator != 0 and numerator / denominator > 2 avoids division by zero because the logical conjunction's second test is evaluated only when the first succeeds. Reversing the tests preserves the abstract truth table for defined operands but can raise a division-by-zero error handled through exception handling. A rewrite involving a side effect must also preserve the needed evaluation behavior.

In positional subtraction, a higher place supplies one unit of its place value when the current place cannot subtract the requested digit. In binary representation, subtraction of one, a low bit toggles at every operation and the borrow continues through each zero bit, stopping at the first one bit. Thus decrementing a binary counter toggles position $i$ exactly when all less significant bits were zero. This is the downward-counting counterpart of a carry propagating through one bits during incrementing.

Use one fixed-size buffered device block per buffer cache key, serialized operations, successful reads and writes, and no outside changes to the backing contents. Record an absent state or a reserved key with validity $v$, dirty flag $D$, and retained-reference count $r$. The bits $v,D$ describe contents, while the nonnegative integer $r$ describes lifetime. Every dirty buffer is valid, so $D=1$ implies $v=1$. Reservation begins at $(v,D,r)=(0,0,1)$; successful fill changes only $v$ to 1. Modifying a valid retained structure sets $D=1$. Releasing a caller changes $r$, not $v$ or $D$.

The first diagram separates contents transitions from reference transitions. Buffer cache writeback temporarily retains the structure while transferring its contents; that internal reference is acquired and released inside the single serialized operation shown. Buffer cache eviction requires a valid clean buffer with $r=0$. The diagram omits read errors, explicit cancellation of invalid reservations, concurrent writes, invalidation and journal ordering. In particular, a journal can forbid otherwise possible buffer cache writeback until its write-ahead logging dependencies are satisfied.

For a small independent check, bound $r$ to $0,1,2$. Starting absent, allow reservation, acquisition, successful fill, modification, successful buffer cache writeback, release of a valid retained structure, and eligible buffer cache eviction. Exactly nine abstract states are reachable: absent, two invalid reservations with $r=1,2$, and six valid states combining $D=0,1$ with $r=0,1,2$. Every transition preserves $D\le v$; the only eviction source has $(v,D,r)=(1,0,0)$. This bounded finite-state machine check supports these safety rules, not a claim that work always completes.

A unit of information; throughout these examples, one byte contains eight bits and is the smallest independently addressed unit of computer memory.

A fixed-sized, aligned block of consecutive bytes transferred and tracked as a unit by a data memory cache. Here cache line and cache block name that unit of data, including its identity in backing computer memory; a resident slot holds a copy. Reading one element can bring neighbouring elements into the same cache line, exploiting spatial locality.

For a nonnegative integer physical address $a$ and a cache line size $B$ bytes, Euclidean division gives the backing cache block number $q$ and offset $d$:Here the brackets denote the floor function and $\mathrm{mod}$ gives the remainder. Cache line $q$ contains bytes at physical addresses $Bq$ through $Bq+B-1$. With $B=16$, read address 52 selects cache block 3, spanning addresses 48–63, and offset 4. Addresses 48 and 52 therefore request the same cache line; addresses 15 and 16 request different cache lines despite being adjacent.

For a cache line size $B=2^b$, the low $b$ bits give the offset. In a fully associative cache, all remaining bits identify the backing cache block as its cache tag; there are no address bits selecting a particular cache slot. An eight-bit illustration gives $52={00110100}_2$: four bits 0011 for cache block 3 and four bits 0100 for offset 4 in the binary numeral system. The eight-bit width is illustrative; the trace does not assume an actual eight-bit machine. ^[ref]

Distinguish this identity from a cache slot, which can hold different cache blocks over time. After read nine in the memory cache example, cache slot 0 contains cache line 3 and cache slot 1 contains cache line 0. Their cache tags identify 3 and 0; neither cache line number is its slot number. Some texts use “cache line” for the storage entry itself; this article reserves cache slot for that role.

A processor status bit recording arithmetic carry. For unsigned addition of $n$-bit inputs, $X+Y=Z+2^{n}C$, where $Z$ is the stored result. In the ARM architecture family, subtraction instead uses $X-Y=Z-2^n(1-C)$, so a set arithmetic carry means no borrow. This convention must be checked for the target instruction set; it is not a signed-overflow test.

For finite alphabets, expanding joint entropy with the chain rule for information entropy givesEach term is mutual information with an entire preceding block, rather than necessarily with one preceding variable. Changing the order changes these individual terms but leaves their sum unchanged.

For the noiseless parity constraint and total correlation, $I(Y;X)=0$ and $I(Z;X,Y)=1$ bit. The conditional block term supplies exactly the dependence that the pairwise comparisons missed. For three identical copies of a fair bit, the two block terms are both one bit. Thus total correlation counts total departure from the product law, not merely the presence of a single irreducible multiway relation.

For finite discrete random variables, the chain rule counts uncertainty in two stages:The joint entropy equals the first variable's information entropy plus the average conditional entropy left in the second. All logarithms here have base two, so the unit is the bit; zero-probability terms contribute zero. The identity holds without assuming independence.^[ref]

Iterating gives $H(X_1,\dots ,X_n)={\sum }_{i}H(X_i\mid X_1,\dots ,X_{i-1})$. The first summand means $H(X_1)$, with no earlier observations. Each later summand counts only the new uncertainty after the preceding random variables are known. For independent random variables, every term is its ordinary information entropy; for dependent ones, adding those ordinary information entropies can count shared uncertainty repeatedly. This is the reason the chain rule for information entropy is useful in lossless source coding and in calculating an entropy rate.

A short proof starts from the factorization of the joint probability mass function into a marginal distribution and a conditional probability:Use this only where $p_{X,Y}(x,y)>0$, then average with those joint probabilities. Summing over $y$ in the first term gives $H(X)$; the second is precisely $H(Y\mid X)$. Repeating the same step with a longer random vector proves the iterated identity. Reversing the order also gives $H(X,Y)=H(Y)+H(X\mid Y)$: the two contributions can change with the order, but their total cannot.

For finite discrete random variables, the mutual information supplied by two observations equals the information supplied by the first plus the additional conditional mutual information supplied by the second:The order changes the individual contributions but leaves the total unchanged.^[ref] To prove the first identity, add and subtract the intermediate conditional entropy:With base-two logarithms, all contributions are in bits. Repeatedly inserting intermediate conditional entropies makes them cancel, as in a telescoping series, and gives the rule for any finite list of observations:where the $j=1$ term is $I(X;Y_1)$. The same calculation with $W$ already known gives $I(X;Y,Z\mid W)=I(X;Y\mid W)+I(X;Z\mid Y,W)$.

For the fair-bit exclusive or example in conditional mutual information, observing $Y$ alone tells nothing about $X$, but observing $Z$ after $Y$ tells everything:Here $H(X\mid Y)=1$ by independence and $H(X\mid Y,Z)=0$ because $X=Y\oplus Z$. In the reverse order, $H(X\mid Z)=1$, so $I(X;Z)=0$ and $I(X;Y\mid Z)=1$: the two observations are individually uninformative but jointly reveal one bit. Simply adding their separate mutual informations would miss all of it.

The chain rule is useful for measuring the value of each new observation without double counting information already supplied. It also guarantees $I(X;Y,Z)\ge I(X;Y)$ by nonnegativity of conditional mutual information. Keeping $Y$ and adding $Z$ can help, even though discarding $Y$ and retaining only a processed $Z$ is constrained by the data processing inequality. Equating the two observation orders gives the precise identitywhich explains why there is no general rule that conditioning decreases mutual information.

Channel capacity is the best reliable information rate available through a specified communication channel. For a discrete memoryless channel with finite input and output alphabets, fixed transition probabilities $W(y\mid x)$, and no input-cost constraint, it isin bits per channel use, when the mutual information uses base-two logarithms. The maximization varies the input probability distribution, while keeping the communication channel fixed. A large input alphabet or a large output information entropy alone does not determine how much information gets through: the receiver needs information about the input, rather than merely unpredictable output.

The operational meaning comes from the Shannon second coding theorem. A block code represents one of $M$ possible messages by a word of length $n$, giving a rate ${\log }_{2}M/n$ bits per use. Take the message label from the uniform distribution on a finite set of those $M$ choices when measuring average decoding-error probability. For every fixed rate $R<C$, there are sequences of such block codes whose decoding-error probability tends to zero as $n$ grows; a rate above $C$ cannot have vanishing decoding-error probability. This is an asymptotic limit, rather than a promise that an arbitrary finite block code at a rate below $C$ works well. See MIT 18.310, Shannon's Noisy Coding Theorem, sections 1–2.

For a binary symmetric channel with crossover probability $p=0.1$, the binary symmetric channel capacity is approximately $0.53100441$ bits per use. Thus a binary code rate $R=1/2$ lies below the limit, while $R=3/4$ lies above it. At $R=1/2$, two transmitted symbols represent one message bit on average; the redundancy helps distinguish messages after noise. This does not split individual physical symbols into fractional pieces.

The result is useful because it separates two tasks. Computing channel capacity establishes the target imposed by the transition probabilities; constructing an effective block code attempts to approach that target with acceptable delay and decoding effort. The surprise is that reliability can improve indefinitely without forcing every positive rate to disappear. In his 1948 paper, Shannon defines noisy channel capacity in section 12 and develops its operational significance in section 13, Theorem 11. His original formulation uses information per unit time; the fixed-duration channel-use convention here expresses the same rate question per symbol. ^[ref].

A deterministic fixed-length channel code consists of a channel encoder $c$ and a channel decoder $d$. It represents $m$ possible messages by length-$n$ input codewords for a communication channel, and estimates the message from the resulting output word. Its code rate is $({\log }_{2}m)/n$ bits per channel use. The maximal decoding error probability requires reliable recovery of every message, while an average error criterion depends on the chosen message probability distribution.

Information about earlier received outputs made available to the channel encoder while choosing subsequent inputs. For repeat-until-success coding over an erasure channel, an erasure report tells the sender to repeat the current bit, while a success report tells it to advance. A forward code rate calculation must specify whether this return communication is supplied freely or included in the resource count.

Two separated players receive uniformly random input bits $a,b$ and output bits $x,y$ without communicating after receiving their inputs. They win when $x\oplus y=ab$, with exclusive or $\oplus$. For deterministic local response functions, the four desired equalities have inconsistent parity, so at most three can hold. Shared randomness only averages such strategies, proving classical winning probability at most $3/4$. In terms of correlations $E_{ab}=\mathbb{E}[(-1{)}^{x+y}\mid a,b]$, the winning probability is $1/2+(E_{00}+E_{01}+E_{10}-E_{11})/8$, directly connecting the game to the CHSH inequality.

Exact quantum teleportation that preserves an arbitrary reference requires at least two classical bits from sender to receiver, even with preshared quantum entanglement. For the essential reduction, encode two classical bits in the four Bell states by Pauli gates on the sender's half of a Bell pair. Teleportation as an identity channel on a reference transfers this half while preserving the encoding. The receiver then has both halves and recovers both bits by a Bell-basis measurement, as in superdense coding.

The classical bit bound itself has a short no-communication theorem proof. For $N$ possible messages and $K$ possible classical transmissions, let ${\sigma }_{m,c}$ be the receiver's subnormalized conditional density matrix before decoding. The no-communication theorem gives ${\sum }_c{\sigma }_{m,c}=\tau$, independent of $m$, so $0\le {\sigma }_{m,c}\le \tau$. For each transmission $c$, let $\{E_{m|c}{\}}_m$ be the decoding POVM. Averaging over equiprobable messages,Perfect decoding of four messages requires $K\ge 4$, hence at least two classical bits. This applies to the preshared resources of the proposed quantum teleportation protocol as well as the Bell pair used for the encoding.

In exact classical identification of an unknown linear Boolean function $f_a(x)=a\cdot x$, a function oracle call returns one bit $f_a(x)$. The query complexity is exactly $n$ for $n\ge 1$. Queries on the standard basis inputs $e_j$ reveal $a_j$, proving sufficiency. For necessity, a depth-$k$ decision tree has at most $2^k$ answer histories and must distinguish $2^n$ hidden binary strings; hence $k\ge n$, even with adaptive inputs. A reversible circuit that preserves $x$ and updates a known target by exclusive or reveals the same one bit per call.

There is also a useful linear algebra proof. Along a fixed history, the chosen inputs form the rows of a matrix $M$ over the two-element finite field, and the replies specify $Ma$. If there are $k<n$ queries, the rank-nullity theorem gives a nonzero vector $v$ with $Mv=0$. Then $a$ and $a+v$ give identical replies throughout this history. This also explains why adaptive choices cannot evade the bound: equal earlier replies force the same later choices.

A designated transition of a clock signal at which a synchronous sequential circuit updates its stored bits. Next-state functions use the state immediately before the edge, and all stored bits update together. This convention is essential when composing circuits: a combinational output computed from the old state can be sampled at that edge, while an additional stored output introduces another cycle of delay.

Keep page frames in a circular list with a reference bit and a scanning hand. Clear set reference bits while scanning; choose the first eligible page frame with a clear reference bit. This second chance approximates LRU replacement with cheaper tracking. When hardware reference flags are absent, revoking a mapping and handling its first later page fault can supply a software reference observation.

For the page-reference trace, number the three page frames 1–3 and initially point the hand at 1. Every access sets the referenced page frame's reference bit to 1. On a miss, scan from the hand: stop at an empty page frame or a resident one with reference bit 0; otherwise clear its reference bit and advance cyclically. Install the requested memory page with reference bit 1 and advance the hand to the next page frame. Hits leave the hand where it is. This fixes a particular variant; clean-victim preference is excluded.^[ref]

After the first three accesses, all reference bits are 1 and the hand is back at page frame 1. Access 4 clears all three reference bits, returns to page frame 1, evicts memory page 1, installs memory page 4, and leaves the hand at page frame 2 with reference bits $(1,0,0)$. This variant gives nine misses for three page frames and ten for four on the displayed sequence. Approximating LRU replacement therefore does not inherit the LRU stack property.

// Destination: computer-science.bigb

Coding theory studies representations of messages that make decoding efficient, reliable, or compact. In channel coding, messages are mapped to separated codewords, so a corrupted word can still be recognized as coming from one particular message. In lossless source coding, a prefix code instead assigns short descriptions to frequent symbols. The objectives differ, but both exploit structure in the set of allowed words.

The simplest error-correction example sends $0$ as $000$ and $1$ as $111$. If the receiver sees $010$, at most one bit flip is consistent with $000$, whereas two flips would be needed to obtain it from $111$. A repetition code therefore recovers the original by taking the majority function of the received bits. The Hamming distance between the valid codewords is three: the spaces of one-error outputs around them do not overlap. This turns error correction into geometry in a discrete space.

The central tradeoff is between code rate and minimum distance. Additional separation makes correction possible but leaves fewer words available for messages at a fixed length. The Hamming bound counts disjoint error neighborhoods, while linear codes turn encoding and error detection into computations with matrices over finite fields. These connections explain why combinatorics, linear algebra, and probability theory all enter the subject.

Richard W. Hamming's Error Detecting and Error Correcting Codes appeared in the Bell System Technical Journal in April 1950, pp. 147–160. Its introduction motivates automatic correction through failures in digital computing, and its constructions go beyond detecting an error to identifying and correcting its location.^[ref]