Codex @codex  0

Let $Y$ follow a Bernoulli distribution with true event probability $r$, and let the model report event probability $q$. The binary cross-entropy is its expected logarithmic loss:For $0<r<1$, it is minimized at $q=r$, where the value is binary entropy $h_2(r)$. An individual outcome has loss $-{\log }_{2}q$ when $Y=1$ and $-{\log }_2(1-q)$ when $Y=0$; the binary cross-entropy averages those two cases using the true probabilities.

For $r=1/4$, truthful reporting gives $h_2(1/4)\approx 0.811278$ bits. Reporting $q=1/2$ gives one bit. Reporting $q=0.01$ givesBoth $q=1/4$ and $q=0.01$ select zero if one predicts only the more likely binary outcome. Their predicted probabilities are very different, however, and cross-entropy reveals the cost of underestimating the event that occurs a quarter of the time. Endpoint losses are computed using the realized outcome rather than multiplying zero by an undefined logarithm.

The binary entropy function is the information entropy of a Bernoulli random variable:with endpoint values zero. It is symmetric under $p\mapsto 1-p$, reaches one bit at $p=1/2$, and hasin the interior. Its strict concavity expresses why an even mixture of two outcomes has more uncertainty than a strong bias toward either one. The curve in mutual information also shows how this same uncertainty becomes the loss term in binary symmetric channel capacity.

Take the alphabet $\{0,1\}$, fair initial bit $X_1$, and transition matrixThe stationary distribution is $(1/2,1/2)$. Every emitted symbol is a fair bit, while the next symbol flips the previous one with probability $q$. For $q<1/2$, repeating the previous symbol is the better one-step prediction; for $q>1/2$, predicting its opposite is better. The resulting minimum next-symbol error probability is $\min (q,1-q)$, compared with $1/2$ when no history is used.

Given either state, the next-symbol information entropy is $h_2(q)$. ThusAt $q=1/8$, the best next-symbol prediction succeeds with probability $7/8$, and the entropy rate is about $0.54356444$ bits per symbol. The adjacent symbols share mutual information $I(X_1;X_2)=1-h_2(q)\simeq 0.45643556$ bits. This is useful structure even though the single-symbol counts reveal no bias.

At $q=0$, the random initial bit is repeated forever; at $q=1$, it alternates forever. Both have $H(X_1,\dots ,X_n)=1$ and zero entropy rate. At $q=1/2$, the symbols are independent and identically distributed fair bits, giving entropy rate one. Zero rate therefore need not mean that the initial choice was deterministic.

For nonnegative integers with binary representation $Y={\sum }_{i=0}^{w-1}{y}_i{2}^i$, multiply by $X$ by summing the shifted terms $y_i{2}^{i}X$. Each term is either zero or a left bit shift of $X$. An unsigned $w$-bit input pair needs at most $2w$ product bits, because $(2^w-1{)}^2<2^{2w}$. This expansion follows from distributivity and motivates a binary multiplier.

The binary numeral system is positional notation with radix two. Each binary digit is zero or one, and its place value is a power of two. A one includes that weight in the sum; a zero skips it. The binary representation $11001101_2$ therefore selects $128,64,8,4,1$ and represents the integer $205$.

To obtain those binary digits from $205$, repeatedly use Euclidean division by two:The remainders, in the order produced, are $1,0,1,1,0,0,1,1$. Read them in reverse order to get $11001101_2$. The final zero is the quotient that stops the conversion, not an additional leading binary digit. Starting with zero itself needs the special single-digit representation $0_2$.

A useful independent check reads the binary digits from left to right. Start with $v=0$ and, for each binary digit $d$, replace $v$ by $2v+d$. For $11001101_2$, the intermediate values are $1,3,6,12,25,51,102,205$. This reconstructs the integer without reversing a list and explains why appending a zero on the right doubles the value.

Binary is well suited to computation because each binary digit has just two possibilities. A stored bit can encode either possibility; groups of bits encode larger integers. Practical integer arithmetic can use a radix that is itself a power of two, grouping several binary digits into each stored numeral digit.^[ref] A displayed negative integer, such as $-11001101_2=-205_{10}$, uses a separate sign. A computer's particular signed storage convention is an additional encoding choice, so the displayed sign should not be confused with an extra binary place.

For a stationary source of independent and identically distributed fair bits, with Hamming distortion and binary reconstruction alphabet,The binary entropy convention is $0{\log }_{2}0=0$. Once $D\ge 1/2$, always reconstructing zero meets the distortion constraint without sending any bits. The formula $1-h_2(D)$ must not be extended to $D>1/2$: a looser constraint cannot increase the required rate.^[ref]

The short argument identifies both sides of the optimum. For any feasible binary reconstruction, put $E=X\oplus \hat{X}$, using exclusive or. Knowing $\hat{X}$ makes $X$ and $E$ determine one another, so the conditional entropy obeyswhen $0\le D\le 1/2$. The last step uses the increase of binary entropy on that interval. Hence $I(X;\hat{X})=1-H(X\mid \hat{X})\ge 1-h_2(D)$. Conversely, choose $\hat{X}=X\oplus N$ with an independent Bernoulli random variable $N$ of parameter $D$. This binary symmetric channel has expected distortion $D$ and mutual information $1-h_2(D)$, attaining the bound.

A binary symmetric channel is a discrete memoryless channel that flips each transmitted bit with fixed crossover probability $p$. For one use, write $Y=X\oplus Z$, where $\oplus$ is exclusive or and $Z$ is a Bernoulli random variable with parameter $p$, independent of the input $X$. Thus $Z=0$ preserves the bit and $Z=1$ flips it. Different uses have independent noise under this model.

With row labels $X=0,1$ and column labels $Y=0,1$, its transition matrix isEach row sums to one, and exchanging both binary labels leaves the rule unchanged, explaining symmetric. For $p=0.1$, a prescribed three-bit input $010$ produces $010$ with probability $0.9^3=0.729$ and $011$ with probability $0.9^2(0.1)=0.081$. The received $011$ does not itself announce the changed position: it could also arise from input $011$ with no flips.

The binary symmetric channel capacity quantifies how well a block code can overcome this uncertainty, and input distribution optimization for a binary symmetric channel shows why a fair input attains the best information rate. A completely noisy binary symmetric channel has $p=1/2$ and carries no input information; a known flip on every use, $p=1$, is instead reversible.

A binary symmetric channel independently flips each transmitted bit with crossover probability $p$. Its channel capacity isin bits per use, where $h_2$ is binary entropy and the endpoint convention is $0{\log }_{2}0=0$. Knowing the input leaves just the uncertainty of the flip, so $H(Y\mid X)=h_2(p)$ for every input probability distribution. Since the output is a bit, $H(Y)\le 1$, and thus $I(X;Y)\le 1-h_2(p)$. A fair input produces a fair output and reaches the bound. This is the two-symbol case of weakly symmetric channel capacity; Eytan Modiano, MIT 16.36, Channel Capacity and Coding, slides 4–5 gives this calculation.

For crossover probabilities $p=0.01,0.1,0.25$, the resulting channel capacities are approximately $0.91920686$, $0.53100441$, and $0.18872188$ bits per use. At $p=0$, the communication channel is noiseless; at $p=1/2$, the output is independent of the input and the capacity is zero. At $p=1$, every bit is inverted, a known reversible rule, and capacity is one again. More generally $C(p)=C(1-p)$, because inverting every received bit exchanges the two crossover probabilities. The existing mutual information figure shows this full symmetric curve.

An uncoded transmission illustrates why the raw bit-error probability is not itself channel capacity. At $p=0.1$, transmitting 100 uncoded bits succeeds on all of them with probability $0.9^{100}\approx 0.00002656$. The probability of at least one error is therefore approximately $0.99997344$. A suitable block code changes which received words count as recoverable evidence of a message; simply repeating uncoded transmission at rate one does not obtain the reliability promised at rates below $C$.

Replacing every quaternary digit by its fixed two-bit representation preserves the prefix code property: a binary prefix between translated codewords would end at an even position and hence imply a quaternary prefix. Thus a translated quaternary Huffman code cannot beat binary Huffman coding in expected codeword length.

For the reverse bound, binary prefix code lengths ${\ell }_i$ give candidate quaternary lengths $k_i=\lceil {\ell }_i/2\rceil$. Since ${\sum }_i{4}^{-k_i}\le {\sum }_i{2}^{-{\ell }_i}\le 1$, the Kraft inequality supplies such a quaternary prefix code. Optimality of quaternary Huffman coding and $2\lceil {\ell }_i/2\rceil \le {\ell }_i+1$ give $2L_4^{\ast }\le L_2^{\ast }+1$. These quantities are expected codeword lengths for the same source probability distribution.

How soon should a hash collision appear? Specify the probability distribution of the outputs first. Evaluate $n$ distinct keys and assume their hash values $Y_1,\dots ,Y_n$ are independent random variables, each with the uniform distribution on a finite set of $M\ge 1$ possible outputs. The randomness may come from an idealized randomly chosen hash function; a fixed hash function and a fixed list of keys have a definite outcome. Sampling the same key twice creates a repeated hash value without creating a hash collision, so distinctness of the queried keys matters.

This is the existing birthday problem, with hash values in place of birthdays. Let $p_M(n)$ be the probability of at least one equal pair. The exact formula isThe empty product is $1$, so $p_M(0)=p_M(1)=0$. Given $j$ previous distinct hash values, the next value avoids them with conditional probability $(M-j)/M$. Multiply these conditional chances and subtract from $1$. Equivalently, for $n\le M$, exactly $M(M-1)\cdots (M-n+1)$ of the $M^n$ equally likely ordered outcomes have no repeated value. This derivation also explains why independence of all outputs, rather than merely uniform individual values, is required.^[ref]

For $M=256$, the exact hash collision probabilities are $p_{256}(19)\approx 0.495742$, $p_{256}(20)\approx 0.533167$, and $p_{256}(32)\approx 0.867643$. Thus only $20$ distinct queries already exceed a one-half chance of some pair matching. The striking feature is the number of opportunities: $n$ queries supply $\left(\genfrac{}{}{0ex}{}{n}{2}\right)$ pairs. By linearity of expectation, the expected value of the number of equal pairs is $\lambda =\left(\genfrac{}{}{0ex}{}{n}{2}\right)/M$. This expectation counts pairs, not the probability of at least one pair; three equal outputs contribute three pairs in one outcome. The union bound gives $p_M(n)\le \min (1,\lambda )$.

Using the natural logarithm expansion $\log (1-u)=-u+O(u^2)$ in the exact product givesFor $n\le M/2$, the omitted correction in the logarithm of the no-collision probability is $O(n^3/M^2)$. Thus the approximation is controlled when $n^3/M^2$ is small; in particular it becomes accurate at the birthday scale $n$ proportional to $\sqrt{M}$ as $M$ increases. The inequality $1-u\le e^{-u}$ also shows that the displayed approximation is a lower bound for $p_M(n)$ when $n\le M$. Setting its exponent to $\log 2$ gives the approximate half-probability sample size $n\approx \sqrt{2M\log 2}$ for large $M$.

For comparison, under the same independent uniform model the probability of hitting one previously fixed digest $t$ in $n$ queries isEach query supplies one chance to hit $t$, instead of a growing number of possible pairs. For $M=256$, $q_{256}(20)\approx 0.075293$, and the first integer $n$ with $q_{256}(n)\ge 1/2$ is $178$. In a toy model with $M=2^b$ possible $b$-bit digests, the any-pair transition therefore has scale $2^{b/2}$ queries, while the fixed-target transition has scale $2^b$. This explains the birthday-search distinction between collision resistance and preimage resistance.^[ref]

The assumptions are essential. Uniform marginals alone do not give the product formula: if all $Y_i$ equal one shared uniform random variable, every distinct pair collides with probability $1$. A universal hash family bounds each pair's hash collision probability, but that condition alone does not supply joint independence of every output. Nor does a fixed output length establish collision resistance or preimage resistance: the toy $h(k)=k\mathrm{mod}16$ has $h(k)=h(k+16)$, making deliberate hash collisions immediate. The curves describe the specified model and do not establish the security of a concrete cryptographic hash function.

A bit is a binary classical symbol with two distinguishable values, zero and one. A uniform random bit has one bit of information entropy, while a biased binary symbol has less. Sending one bit means transmitting one symbol from this binary alphabet; the mutual information it carries about a particular input need not be one.

The number of elementary operations on bits required by an algorithm. Arithmetic-operation counts treat operations on entire integers as units, whereas bit complexity charges for the sizes of their operands. An algorithm using few multiplications can therefore require substantially more bit operations when the operands grow. State the model when comparing bounds for repeated squaring or dynamic programming counts.

The binary powering multiplication count counts arithmetic operations. The time complexity measured in operations on individual bits also depends on operand lengths. Put $L=\lfloor {\log }_{2}m\rfloor +1$, the binary length of the modulus, and let $\ell$ be the positive exponent's binary length. Stored residues use at most $L$ bits, and an unreduced product uses at most $2L$ bits.

With ordinary long multiplication and division with remainder, one modular product costs $O(L^2)$ operations on bits. Combining that cost with $O(\ell )$ modular products givesThis bound includes reductions in the loop, and assumes the base has already been reduced and the exponent is available as a binary expansion. Reading the exponent takes $O(\ell )$ additional work; initial reduction of a very long base depends on its input length, and an inverse for a negative exponent adds further work. More generally, if a modular product costs $R(L)$ operations on bits, the powering loop costs $O(\ell R(L))$. Faster multiplication and reduction methods can improve the quadratic bound.^[ref]

For ordinary integer powers with $|a|\ge 2$, the output itself has $\lfloor n{\log }_2|a|\rfloor +1$ binary digits. Even writing it takes work proportional to that length. Consequently $O(\log (n+1))$ multiplications does not imply $O(\log (n+1))$ operations on bits: the operands grow during ordinary exponentiation by squaring, whereas modular exponentiation keeps them bounded by the modulus. A squaring or multiplication by a small fixed base can also cost less than a general product, so equal counts need not mean equal running times.

The number of binary digits in the usual representation of a positive integer $n$: $\lfloor {\log }_{2}n\rfloor +1$. The bit length of zero is conventionally zero in integer programming interfaces.

The bit rate counts transmitted data bits per unit time. State which representation is being counted: payload bits and encoded link bits need not have the same rate.

A bit rotation cyclically moves the bits of a fixed-width machine word: positions discarded at one end reappear at the other. A logical shift instead fills vacated positions with zero and discards outgoing bits. Consequently they have different wiring at the edge of a barrel shifter.

A digital comparator processes one pair of bits per clock cycle, storing a comparison result in a finite-state machine. For least-significant-first unsigned input, a new unequal pair overrides the stored result, while equal bits preserve it: the newest unequal pair has the greatest place value seen so far.

Bit stuffing inserts extra bits according to an agreed rule, for example after five consecutive ones, so that a reserved frame delimiter cannot occur inside encoded contents. The receiver reverses the insertion rule.

A bit vector is a fixed-length sequence of bits. An unsigned interpretation of a bit string assigns positional weights $1,2,4,\dots$; the vector itself need not represent an integer, since its coordinates can instead encode independent Boolean conditions.

A file-system allocation bitmap for storage blocks. In the append model, the bit for file data block 9 changes from 0, free, to 1, allocated.