Algorithm design 2026-10-31
Constructing an algorithm that meets a stated specification. Choices such as divide-and-conquer algorithms, dynamic programming and greedy algorithms organize the construction; an algorithm correctness argument establishes that the result meets the specification.
Dynamic programming Created 2026-09-24 Updated 2026-10-20
Dynamic programming solves a mathematical optimization problem by choosing reusable subproblems, recording their value functions, and combining already solved cases through a recurrence relation. Its attraction is that many possible decision histories can share one continuation calculation. A dynamic programming state must retain everything needed to determine the allowed future choices and their rewards. The 0-1 knapsack problem, for example, needs both the available capacity and which items remain available; capacity alone would lose the restriction that each item can be used at most once.
Two ideas explain when this works. Optimal substructure makes an optimal continuation valid inside a larger optimal solution. When subproblems overlap, storing each answer avoids repeated work: memoization computes entries as needed, while a table fills them in dependency order. Start from known base cases. For a problem indexed by increasing size, compute smaller cases first; for a finite sequence of future decisions, work backwards from the terminal reward, as in the Bellman equation. See Bertsekas, MIT lecture 2: optimality and state augmentation.
Consider a 0-1 knapsack problem with item weights , item values , and capacity . Define the value function to be the largest total item value obtainable from the first items with total weight at most . Here and specify the dynamic programming state. All weights and capacities are nonnegative integers. The empty selection supplies the base case . The recurrence relation isThe two candidates skip or take item . Both use the previous row, so taking item cannot take that same item again. The condition prevents a negative remaining capacity; no entry with is needed. The optimal substructure is concrete: after taking item , its companions must solve the smaller 0-1 knapsack problem on the first items with capacity . Otherwise replacing them by a better feasible selection would improve the whole selection. Compare [ref]
https://ocw.mit.edu/courses/6-006-introduction-to-algorithms-fall-2011/95e744ee64464cb6bc8f8264faeea080_MIT6_006F11_rec21.pdf. MIT 6.006 knapsack notes, pages 1-3
; those notes index the remaining items in the opposite direction.The full computation is small enough to see at once:For example, and . Choosing the feasible item with largest individual value takes weight , earns , and leaves capacity , which cannot fit another item. This greedy algorithm misses the two-item selection of weights and , whose value is . This example tests the largest-individual-value rule; choosing by value divided by weight happens to find the optimum here.
A knapsack table builds the optimal value from smaller subproblems
. The value function of this 0-1 knapsack problem is computed row by row. Outlined entries show the reconstruction of an optimal selection: skip weight 4, take weight 3, then take weight 2. The total weight is 5 and total value is 7. Details.The number is the optimal value, while the choices attaining it form a control policy. To reconstruct one optimal selection, start at . Equality with allows skipping item . At , taking item gives , so continue to . Take item and finish at . In a tie, either maximizing choice suffices to recover one optimum. Recording choices during the calculation or comparing stored entries afterwards recovers the control policy; a final optimal value alone does not describe the selection.
Knapsack dependencies and one optimal reconstruction
. Each node is a dynamic programming state labeled by its value function. Arrows point to the earlier-row entries used in the recurrence relation, forming a Directed acyclic graph. Dashed arrows skip an item; blue arrows take it. The orange graph path reconstructs the optimal selection with value 7. Compute in the opposite direction to the arrows, beginning at the zero base row. Details.For items and integer capacity , the table uses time complexity and space complexity , counting arithmetic operations and stored numbers using Big O notation. This is pseudo-polynomial time: when weights and capacity use the binary numeral system, the capacity needs only digits. A capacity near creates about columns despite needing only about digits. Thus this algorithm is practical for moderate capacities but does not establish polynomial time in the encoded input length. See MIT 6.006 recitation 19: knapsack and input size. For the optimal value alone, one array reduces space complexity to : for each item, update capacities in descending order, from down to . Ascending order would read entries already updated for that item and could reuse it. Reconstruction then requires extra information or recomputation.
The link between the knapsack problem and dynamic programming is already explicit in a 1957 paper by Richard Bellman at RAND: its abstract discusses an exact knapsack problem solution using functional equations and cites his 1957 book. See Bellman, Comment on Dantzig's Paper on Discrete Variable Extremum Problems (1957).
Finite greedy basis extension and choice 2026-10-14
For an explicit basis extension in , start with , where . Its linear span is the line . Adding preserves linear independence, because is outside that line, and gives a basis. Every vector has the finite linear combinationThe vector cannot be added while preserving linear independence: it is already in the linear span. Many choices outside the line work, so maximality does not select a unique basis.
For a finite-dimensional vector space with a supplied finite spanning set, a greedy algorithm scans the list, retaining each vector precisely when it is outside the linear span of those already retained. Starting with a linearly independent , the same scan adjoins only the needed vectors. Retained vectors remain linearly independent, and every skipped vector is in the final linear span, so the result is a basis containing . Finite reasoning needs no axiom of choice.
For an arbitrary vector space, this finite stopping argument is unavailable. Accepting Zorn's lemma asserts existence of a maximal element once every chain in a partial order has an upper bound; it supplies no general computational recipe. Over ZF, Zorn's lemma, the axiom of choice and the well-ordering theorem are equivalent. Particular infinite vector spaces can still have explicit bases, such as for the vector space of univariate polynomials over a specified field; infinity alone does not force a nonconstructive argument.[ref]
How to use Zorn’s lemma by Timothy Gowers. How to use Zorn’s lemma, “Further general remarks”, immediately after Example 1 continued: equivalence with choice and well-ordering
Graphic matroid 2026-10-14
For a finite undirected graph, the graphic matroid has its edges as the ground set. A subset is a matroid independent set when its edges form a forest on the original vertices, including isolated vertices. Its matroid bases give a spanning tree in each connected component of a graph; for a connected graph, each matroid basis is a spanning tree.
For a graph triangle on vertices , label the edges , , and , with weights . The empty set, all singletons, and all pairs are forests; the triple contains the graph cycle . Thus its eight subsets have exactly the independence pattern checked in the weighted three-vector matroid, and the same fifteen exchange comparisons pass. The matroid greedy algorithm chooses , chooses , and rejects , producing the heaviest spanning tree, of weight .
A weighted graph triangle admits greedy selection of a matroid basis; a hereditary independence system without exchange traps a greedy algorithm at weight instead of
. Details. Exchange outline. Deleting edges cannot create a graph cycle. If forests have , some edge of joins different connected components of a graph of : otherwise, within each such component, would have at most its number of vertices minus one edges, giving . Adding that edge keeps a forest. This is the matroid independence axioms expressed through connectivity.
Greedy algorithm Created 2026-10-13 Updated 2026-10-20
A greedy algorithm builds a feasible solution by repeatedly committing to a locally preferred next choice. Each commitment is final: the algorithm does not revisit it when later choices become available. In mathematical optimization, the important question is whether those commitments can still attain the best objective function value.
For example, a shared telescope can accept several observation requests with fixed time slots. In interval scheduling, every completed request counts equally, and overlapping requests cannot both be accepted. Choosing the request that finishes earliest leaves the resource available sooner. The example below turns that intuition into a precise rule, then shows why choosing the earliest start or shortest duration can lose jobs.
Greedy algorithms are attractive because a single pass can replace a search through many candidate subsets. However, "locally best" is only a design proposal. A problem-specific greedy-choice property, often established by an exchange argument, must connect the local decision to the actual objective function. The largest-individual-value rule in the 0-1 knapsack problem fails; Dijkstra's algorithm needs its nonnegative-edge assumptions. A rule's success in one optimization task does not justify it in another.
Greedy coin change for powers of an integer 2026-10-31
For an integer and , unlimited denominations admit optimal greedy algorithm. In a minimum-count solution, no smaller denomination can occur times: carrying those stamps to the next denomination reduces their count by . Thus all counts below the maximum are base- digits, whose total value is at most . The maximum-denomination count is forced to be the quotient on division by , and repeating the argument forces every greedy remainder step.
Greedy Egyptian fraction expansion 2026-10-25
For a positive proper residual fraction , with , choose , append the unit fraction , and replace the residual by in lowest terms. This is a greedy algorithm: the chosen unit fraction is the largest one not exceeding the residual.
The inequalities imply . Thus the numerator of the reduced residual strictly decreases while positive, proving algorithm termination by a nonnegative-integer loop variant. The loop invariant is that the accumulated sum plus the residual is the original rational number. Since and , the new residual is less than . If it remains positive, its next chosen denominator is therefore strictly greater than ; this expansion also satisfies the distinct-denominator convention. The proof does not bound the denominators by the input's fixed-width range. Arbitrary-precision arithmetic, checked conversion, or a different representable expansion can therefore be necessary in code.
Greedy matching 2026-10-14
A greedy matching starts with the empty matching in a graph, scans all edges in some order, and selects an edge exactly when both endpoints are unmatched vertices in a matching. For a finite graph, a complete scan returns a maximal matching: a rejected edge had a matched endpoint, which stays matched thereafter. Stopping before all edges have been considered need not give maximality. Changing the order can change the result; on P₄, middle-first gives one pair, whereas outer-first gives two. A greedy algorithm of this form never revisits its choices, while an augmenting path in a matching does.
Hereditary nonmatroid greedy obstruction 2026-10-14
On the ground set , permit exactlyThis lists every admissible subset; the excluded subsets are , , and . It is an independence system: deleting elements from produces its listed singletons or the empty set, and deleting from a singleton produces the empty set. But and violate exchange: neither nor can be added to . Thus it is not a matroid.
The descending-weight greedy algorithm admits first, then rejects both and , returning with weight . Exchanging the tied order of changes nothing. The admissible pair weighs , exceeding every singleton and the empty set, so it is the true optimum. Both and are maximal elements under inclusion of subsets, but only the pair has maximum cardinality. This identifies exactly where the matroid greedy algorithm loses its guarantee: heredity preserves admissibility under deletions, while exchange prevents a locally maximal selection from being too small.
Huffman coding Created 2026-09-24 Updated 2026-10-14
Huffman coding makes frequent symbols cheap to store by giving them short codewords, while allowing rare symbols longer codewords. For a known probability distribution on a finite alphabet with at least two positive-probability symbols, binary Huffman coding constructs a prefix code minimizing expected codeword length among binary prefix codes that encode one source symbol at a time. The objective is an average under that probability distribution; it does not promise to shorten every message. Encoder and decoder must agree on the resulting Huffman code. [ref]
The construction grows a binary tree from its leaves towards its root of a tree. Start with one leaf per symbol, weighted by its probability. Repeatedly join the two currently smallest weights beneath a new parent in a rooted tree, whose weight is their sum, and replace the pair by that subtree. Include newly formed subtrees when choosing the next pair. When one tree remains, label the two outgoing edges at every internal vertex with and ; the labels on the graph path from the root of a tree to a symbol give its codeword. This is a greedy algorithm: each step makes a locally cheapest merge, yet the result is an optimal prefix code.
For a concrete example, use probabilities , , , . First join and , creating weight . The remaining weights are for , for the subtree, and for . Join the two weights to make , then join with that subtree to make . The merge weights are therefore . Each merge adds one bit to every codeword below it, so their sum is the expected codeword length: bits per source symbol. The diagram fixes the edge labels to produce the Huffman code used below.
The useful insight is that choosing short codewords is a resource allocation problem: the prefix code condition constrains the available codeword lengths through the Kraft inequality. Huffman coding solves this one-symbol problem exactly for the supplied probability distribution. For repeated emissions of a memoryless classical information source, the same Huffman code can be applied symbol by symbol. More general compression can exploit dependencies or encode groups of symbols; the one-symbol optimality statement does not compare all such methods.
Independence system 2026-10-14
An independence system consists of a finite set and a nonempty family of subsets such that and imply . This is the hereditary condition: deleting chosen elements keeps a selection admissible. In particular, the empty set belongs to . An independence system becomes a matroid when it also satisfies the matroid independence axioms for exchange. The hereditary nonmatroid greedy obstruction shows why deletion alone does not justify a greedy algorithm.
Integer logarithm 2026-10-31
For integers and , the integer logarithm is the largest nonnegative integer such that . Equivalently,The equivalence follows from the monotonicity of exponentiation in its exponent and the definition of the floor function. For example, , so the base-three integer logarithm of is .
A multiplication loop maintains and accepts another factor while . Repeated squaring makes the search faster: precompute starting at , and consider these factors in decreasing . Include a factor when the accumulated product remains at most . If , the loop invariant before considering is . Accepting exactly when halves that uncertainty, proving the greedy algorithm returns . This takes arithmetic operations; it is not the same claim as this bound on bit complexity.
For positive operands in bounded integer arithmetic, test before multiplying by . Integer division makes this equivalent to while preventing integer overflow in a rejected candidate.
Interval scheduling 2026-10-18
Given a finite set of jobs with fixed half-open intervals , where , choose a subset of maximum cardinality. Each job occupies one shared resource throughout its interval. A feasible solution has pairwise disjoint intervals: for distinct selected jobs , either or . Thus a finish and the next start may coincide. Every job has value one; occupied time is not the objective function.[ref]
Lecture 1: Introduction. 6.046J Design and Analysis of Algorithms, Spring 2015; Printed p. 2, Interval Scheduling: fixed requests, one resource, compatibility with endpoint equality, maximum size
Consider these six original jobs, with arbitrary time units:Job can be followed immediately by and . Job conflicts with all three. Starting the resource earlier is therefore not automatically beneficial: one long commitment can displace several shorter requests.
The earliest-finish greedy algorithm uses a sorting algorithm to put jobs in nondecreasing finish time. Scan that order, accepting the first job and then each job whose start is at least the last accepted finish. Break finish-time ties arbitrarily; an empty input returns the empty set. For these jobs, the order and decisions areAfter , job starts before the current finish five and is rejected. Accept ; then reject because six is before seven. Finally accept . The result is , with cardinality four. Leaving idle makes room for more jobs.
By contrast, the earliest-start rule scans . It accepts , skips , accepts , and skips . Its feasible solution contains only two jobs. Here and meet at time six and are compatible; and overlap on .
Earliest finish accepts four jobs; earliest start accepts two
. The same six original intervals appear in both panels. Colored jobs are accepted and gray jobs rejected. A filled start and hollow finish depict a half-open interval; jobs meeting at an endpoint are compatible. Details.There is also an elementary optimality certificate for this example. The four subsets , , and form a partition of a set of the jobs. A feasible solution takes at most one job from each group: overlap, as do , and each other group is a singleton. Hence four is an upper bound, attained by . Checking all subsets gives a unique optimum of size four: .
The earliest-finish theorem states that this greedy algorithm maximizes job count for every finite instance of interval scheduling. For the exchange argument, let be the first job in an optimal feasible solution and an earliest-finishing job. Every later selected job satisfiesReplace by ; compatibility and cardinality survive. The remainder must be optimal among jobs starting at or after , or improving it would improve the whole selection. Mathematical induction on the smaller remaining instance, starting with the empty set, completes the outline.[ref]
Greedy Algorithms. Algorithms by Jeff Erickson; §4.2 Scheduling Classes, Lemma 4.3 and Theorem 4.4, printed pp. 163–164; endpoint equality is allowed by our half-open convention
Why is the algorithm fast? Once sorted, each job is examined once, using only the last accepted finish to decide compatibility. The scan takes time; comparison sorting algorithms give total time complexity for , in Big O notation, assuming constant-cost endpoint comparisons. Already sorted input needs only the scan.[ref]
Lecture 1: Introduction. 6.046J Design and Analysis of Algorithms, Spring 2015; Printed pp. 2–4, earliest-finish rule, compatibility and first-job exchange; scan cost follows by counting one comparison per subsequent request
Interval stabbing problem 2026-10-31
Given finitely many nonempty closed intervals , choose as few points as possible so that each interval contains a chosen point. An optimal greedy algorithm repeatedly chooses the smallest end of an uncovered interval. After choosing , any still uncovered interval starts strictly after , since its end cannot be smaller than . Thus the intervals whose ends trigger successive choices are pairwise disjoint. Every feasible point set needs a separate point in each triggering interval, proving that the greedy point set is optimal. Sorting by end and scanning takes time. The inclusion of endpoints means an interval starting at the last chosen point is already covered.
Kruskal's algorithm Created 2026-10-14 Updated 2026-10-18
Kruskal's algorithm constructs a minimum spanning tree of a finite connected graph with undirected edges and real edge weights. Start with all vertices and no edges, a forest of isolated vertices. Sort the edges by nondecreasing weight, then accept an edge exactly when its endpoints belong to different current connected components of a graph. Otherwise reject it: the existing graph path between its endpoints would close a graph cycle. Each acceptance merges two connected components of a graph. Stop after acceptances, when the forest has become one spanning tree.
This is a greedy algorithm for economical network connection: only the next affordable connection and the current connected components of a graph matter. Unlike Prim's algorithm, which grows one reached tree, Kruskal's algorithm can grow several separate trees before joining them. The essential safeguard is avoiding a graph cycle, rather than simply buying the cheapest fixed number of edges.
Joseph Bernard Kruskal described this rule as Construction A on page 49 of his February 1956 paper, On the shortest spanning subtree of a graph and the traveling salesman problem; page 50 gives an exchange argument for distinct weights.[ref]
https://langevin.univ-tln.fr/cours/graphe/bonus/Krukskal.pdf. Joseph B. Kruskal, Jr., Proceedings of the American Mathematical Society 7(1), February 1956, pp. 48–50; inspected scan, Construction A and final exchange argument
Lagrange's four-square theorem 2026-10-13
Every nonnegative integer is a sum of squares with four terms, with zero terms allowed. Precisely, for every integer , there exist nonnegative integers such thatAllowing arbitrary integer roots gives the same statement: changing a root to its absolute value leaves the resulting square number unchanged. Four is a uniform upper bound on the number of nonzero terms, rather than a demand for four positive terms. Repeated square numbers are allowed. Conrad states the positive case and explicitly explains the role of zero terms; follows by taking all four roots zero. [ref]
https://kconrad.math.uconn.edu/blurbs/ugradnumthy/descent.pdf. Keith Conrad, Proofs by Descent, Section 6, opening paragraph and Theorem 6.6
The small cases make the convention visible:In particular, could not be a sum of squares with four strictly positive roots, since each term would be at least one. Geometrically, a square of integer side length has area : the theorem says that any nonnegative whole-number total area can be split into at most four such square areas. It does not assert that those squares tile one larger square or a prescribed rectangle.
Why does the fourth term matter? Two square numbers do not suffice for : in modular arithmetic, a square number has remainder zero or one modulo four, so their sum cannot be three modulo four. The sum of two squares theorem gives a complete criterion for the positive integers representable with two terms. Even three terms leave gaps. A square number has remainder or modulo eight; adding three of these, with repetition allowed, never gives seven modulo eight. Thus cannot be sums of three square numbers, although Lagrange's four-square theorem represents all of them with four. This also proves that the bound four cannot be reduced to three.
The modulo-eight obstruction is not the only way that three terms can fail. For example, needs four: if three square numbers added to , their remainders modulo four would force all three roots even. Dividing by four would then express as a sum of three square numbers, contradicting the preceding calculation. These short checks explain the examples without requiring a full classification of sums of three square numbers.
The practical benefit of Lagrange's four-square theorem is an existence guarantee for a family of Diophantine equations: for every nonnegative integer , the displayed equation has an integer solution. It does not specify a unique solution. For instance, . To find a representation by a finite search, enumerate and test the sum. The floor function supplies a finite bound because no single term can exceed , and sorting roots removes duplicate orderings without discarding any possible sum. The theorem guarantees that the search succeeds.
A greedy algorithm that repeatedly subtracts the largest available square number need not find four terms: for it chooses , while uses four. The appeal of the theorem is this gap between local choices and a global promise: individual square numbers become farther apart, and two or three terms leave arithmetic obstructions, yet four terms cover every nonnegative integer.
For historical orientation, Conrad dates Lagrange's four-square theorem to and identifies the memoir as Démonstration d'un théorème d'arithmétique: it appeared in the Berlin Academy's volume for the year , published in , pages –. Keeping the volume year and publication year distinct avoids an apparent conflict between dates. [ref]
https://kconrad.math.uconn.edu/blurbs/ugradnumthy/descent.pdf. Keith Conrad, Proofs by Descent, Theorem 6.6 and reference 11
Maximal matching Created 2026-09-24 Updated 2026-10-14
A maximal matching is a matching in a graph to which no extra edge can be added while retaining all current selected edges. Equivalently, no edge joins two unmatched vertices in a matching; those vertices form an independent set. Every maximum cardinality matching is a maximal matching, but the converse fails.
Take the path graph with vertices and edges . A greedy algorithm that first selects the middle edge produces . Both remaining edges touch a matched vertex, so this size-one matching in a graph is maximal. Yet has two edges. No matching in a graph on four vertices has more than two edges, so is a maximum cardinality matching. The two remaining free vertices of , namely and , are not adjacent; that does not prevent a longer augmenting path in a matching from joining them.
Choosing the middle edge of P₄ gives a maximal matching of size one; the outer edges give a maximum cardinality matching of size two
. Details. Maximality is useful as a quick stopping rule and a modest guarantee. For any maximal matching and maximum cardinality matching in a finite simple graph, every edge of touches an endpoint of an edge of . Assign it to one such edge of . Since has disjoint endpoints, each edge of receives at most two assignments, proving . The P₄ example reaches equality. Thus maximality guarantees at least half the largest cardinality, but says nothing comparable about arbitrary edge benefits. Its selected endpoints also form a vertex cover, as explained under vertex cover.
A greedy algorithm builds a feasible solution by repeatedly making the locally preferred available choice, committing to that choice without reconsidering earlier choices. “Preferred” must be specified by the problem's rule, such as choosing the cheapest eligible edge.
Local preference alone does not prove global optimality. A correctness argument needs a property ensuring that some optimum extends the choices already made, often proved by an exchange argument. Thus the distinguishing rule is
A divide-and-conquer algorithm splits an instance into smaller instances of the same kind, solves them recursively, and combines their answers. The smaller instances should be sufficiently smaller to ensure algorithm termination; the combination cost determines whether splitting saves work. The two halves of merge sort illustrate this separation.
Dynamic programming identifies a collection of subproblems and dependencies among them, solves each distinct subproblem once, and stores its answer. A table in dependency order or memoization prevents repeated evaluation of overlapping subproblems. For optimization, an optimal substructure argument justifies constructing an optimal answer from optimal smaller answers.
A greedy algorithm repeatedly makes a locally best feasible choice and commits to it. It avoids exploring alternative continuations. Algorithm correctness therefore needs a greedy-choice property, often proved by replacing the first choice of an optimal solution with the greedy choice without worsening it.
Divide-and-conquer algorithms suit problems with largely independent, balanced subproblems and an inexpensive combination step. In merge sort, the two halves are independently sorted and merged in linear time. Its recurrence relation is , giving time complexity.
Dynamic programming suits overlapping subproblems with a compact state description. For the coin change problem, let be the minimum number of coins needed to make amount . With positive denominations containing ,Every optimal nonempty answer has a last coin and an optimal answer for ; otherwise replacing the remainder improves it. Filling takes time complexity and space complexity. Repeated amounts are computed just once. This bound is in the numeric amount , not its binary input length.
A greedy algorithm suits problems where one can prove a greedy-choice property. For unweighted interval scheduling, choose an interval with earliest finishing time, discard overlapping intervals, and repeat. Among optimal schedules, replace the first interval by the earliest-finishing one. The replacement finishes no later, so every later interval remains feasible. Applying the same argument to the remaining intervals proves optimality. After sorting by finish time, a single scan suffices.
These examples separate the relevant structural reasons: independent halves, repeated amount states, and an exchange argument certifying each irreversible choice.








