Write and . Suppose each vertex identifier, list descriptor, link and optional edge weight occupies a constant number of machine words, and indexed array access and identifier comparison take constant time. There are list descriptors and directed, or undirected, neighbour entries. Hence adjacency lists use machine words in Big O notation. A full adjacency matrix has entries, regardless of . These are counts under a storage model, not fixed byte sizes.[ref]
Our undirected example has six list descriptors and fourteen neighbour entries, versus thirty-six full adjacency matrix entries. The numbers and count different kinds of logical items: they do not prove a particular byte saving. A linked list also needs links; an array may have unused capacity. Storing only half of a symmetric matrix changes its exact count while leaving quadratic growth. If identifiers or weights require growing numbers of machine words, their size must be included in the space complexity.
Figure 1.
Adjacency matrix storage and queries compared with adjacency lists on the same six-vertex graph
. Details.
An unsorted adjacency list answers "does the edge exist?" by linear search through . In the worst case it inspects all entries, giving time, or for . Even if is absent, an uninspected entry could still contain it, so arbitrary lists require the whole scan. On our example, looking for in 's list compares with and fails after three comparisons. A full adjacency matrix instead answers from its single entry in time.[ref]
Different promises change the query. A sorted array of neighbour identifiers permits binary search in time, but maintaining order after insertion may require shifting entries. A per-vertex hash table permits expected constant-time membership under the hashing and load assumptions of that data structure; it does not give that guarantee to an arbitrary adjacency list. A stored list length can likewise make a vertex degree query constant-time even though neighbour enumeration still has to visit its entries.
The advantage grows with scale. For a graph path on vertices, , so there are undirected neighbour entries and descriptors, versus full adjacency matrix entries. Such a family is a sparse graph with . For a complete graph, , so the undirected neighbour entries already number : the adjacency list advantage in growth rate disappears. Choose by the workload: following existing edges favours adjacency lists; many arbitrary pair-membership queries can favour a full adjacency matrix. The representation organizes the same graph for different questions.
Breadth-first search Created 2026-09-24 Updated 2026-10-14
Breadth-first search is a graph search that explores successive layers of graph distance from a starting vertex . Here the graph is an undirected graph on a finite set of vertices, and distance counts edges, with no differing edge weights. Mark as discovered, assign , and place it in a FIFO queue. Repeatedly remove the front vertex and scan its adjacency list. Whenever a graph neighbour is still undiscovered, mark it immediately, record as its parent in a rooted tree in the discovery tree, assign , and enqueue at the rear. Marking on insertion ensures that another edge cannot enqueue the same vertex again. [ref]
Use the same undirected graph as depth-first search, with vertex set and edges
where denotes the unordered pair . Start at and scan graph neighbours in lexicographic order of their one-letter labels. The diagram's generator computes the following layers and the parent in a rooted tree assigned to each discovered vertex; the source is the root of a tree and has no parent in a rooted tree.
Graph distance from Vertices in discovery orderParent in a rooted tree in the same order
none
Figure 1.
Breadth-first search: graph distance layers and two ordered FIFO queue snapshots
. Solid edges record first discovery; dashed edges are still present in the original graph. The FIFO queue contains after processing and after processing , with its front on the left. Colors encode graph distance in the graph without edge weights. Details.
The discovery and removal orders both happen to be . After processing , the FIFO queue is : must be processed before the newer . Processing then skips the already discovered and enqueues , giving . Processing adds nothing. Processing discovers , giving ; later sees already marked and leaves its parent in a rooted tree unchanged. The discovery edges are , forming a rooted tree on all seven vertices even though the original graph contains graph cycles. The discovery tree contains six edges; it need not contain every examined edge.
First arrival is shortest because the FIFO queue keeps assigned distances in nondecreasing order, with entries from at most two consecutive layers. While processing distance , newly discovered vertices receive distance and join the rear. If an undiscovered vertex had a shorter graph path, its preceding vertex on that graph path would lie in an earlier layer and would already have discovered it. Thus is the minimum number of edges from to . This is a mathematical induction over the layers, not an assumption about the drawing. Following the parent in a rooted tree backwards at each step gives ; reverse that sequence to obtain the three-edge graph path . is another shortest graph path, but the specified graph neighbour order makes discover first. Changing that order can change the parent in a rooted tree without changing graph distances. [ref]
The shortest-route guarantee concerns the shortest path problem measured by edge count. It also minimizes total cost when every edge has the same positive cost. With unequal edge weights, first discovery need not minimize total cost; nonnegative costs can instead be handled by the Dijkstra algorithm. Breadth-first search is useful for minimum-hop routes and for finding all vertices within a specified graph distance. Its FIFO queue finishes an older layer before the next one, whereas depth-first search uses a call stack or explicit LIFO stack to pursue one branch immediately.
With adjacency lists and constant-cost discovery checks, together with constant worst-case or amortized analysis cost for FIFO queue insertion and removal, the time complexity is in Big O notation. Every reached vertex is enqueued and removed once, each incident edge is scanned at most twice in an undirected graph, and initializing all marks and distances takes . The FIFO queue contains at most vertices; the example's maximum is three. Discovery marks, distances and the recorded parent in a rooted tree for each nonroot vertex also give auxiliary space complexity, excluding the graph. An array that shifts every entry on front removal can lose the linear time complexity; the FIFO queue abstraction alone does not make removal constant-cost. Ordering the adjacency lists is a separate preprocessing cost.
A single start reaches only its connected component of a graph. Unreached vertices retain distance and have no recorded parent in a rooted tree. Restarting at each still-undiscovered vertex, while preserving all discovery marks, produces a forest covering the graph; each new root of a tree gets distance zero, so these distances refer to that root of a tree, not to the original .
Depth-first search 2026-10-13
Depth-first search is a graph search that follows a branch as far as it can before returning to try the next unexplored branch. Starting at a vertex of an undirected graph on a finite set of vertices, mark as discovered. Scan its graph neighbours in a fixed order: for each undiscovered vertex, record the current vertex as its parent in a rooted tree in the discovery tree, mark the new vertex, and explore it immediately using recursion. Return when its adjacency list is exhausted. Discovery marks persist after return, preventing a graph cycle from causing repeated exploration. This branch-and-return rule is the maze exploration described in MIT's notes. [ref]
Use the same unweighted undirected graph as breadth-first search, with vertex set and edges
where denotes the unordered pair . Visit graph neighbours in lexicographic order of their one-letter labels, starting at . The discovery order, computed by the diagram's generator, is
The discovery edges in order are . They form a rooted tree spanning the reached vertices, with root of a tree . The other edges, and , remain in the original graph but never discover a new vertex. In particular, the graph cycle causes no repeated discovery.
Figure 1.
Depth-first search: discovery edges and the active call stack on seven vertices
. Numbers show first-discovery order. Solid edges form the discovery tree; dashed edges belong only to the original graph. The right panel shows the call stack from oldest call to current call, including the return from before entering . Details.
At there is no undiscovered graph neighbour, so its call returns to . The next call enters , then , then , then . Immediately after discovering , the call stack is , with on top; is still discovered but its call has finished. The maximum call stack depth is six, although seven vertices are discovered. Following the recorded parent in a rooted tree backwards at each step reconstructs a graph path; here the route from to is , using five edges. Breadth-first search instead finds , using three edges. Depth-first search establishes reachability but does not guarantee the fewest edges.
An explicit LIFO stack can replace recursion, but its entries and discovery rule must be specified. To reproduce this exact traversal, store one frame per active vertex, retaining the next position in its adjacency list. Advance the top frame by one graph neighbour; if that vertex is undiscovered, mark it and push its frame immediately; pop an exhausted frame. Pushing all undiscovered graph neighbours at once and marking them on push is a different iterative variant. Even pushing them in reverse lexicographic order does not generally reproduce the parent in a rooted tree relationships above: would already be marked while processing , so could not discover .
With an adjacency list representation, constant-cost discovery checks and frame operations, and the graph neighbours already ordered, depth-first search has time complexity in Big O notation. Each reached vertex is entered once and each incident edge is examined at most twice in an undirected graph; initializing all discovery marks costs . The call stack has depth at most , since no active vertex repeats. Discovery marks and the recorded parent in a rooted tree for each nonroot vertex also need auxiliary space complexity, excluding storage for the graph. Sorting adjacency lists to obtain a chosen order is additional work. [ref]
One start explores only its connected component of a graph. To cover a disconnected graph, start again at every still-undiscovered vertex while keeping all discovery marks; the discovery trees form a forest. Depth-first search is useful for reachability, constructing a spanning tree of a connected graph, and organizing the return steps needed by a depth-first traversal of a tree. For detecting a graph cycle in a simple graph that is an undirected graph, an edge to an already discovered vertex other than the current parent in a rooted tree closes a graph cycle; the immediate return edge to that parent in a rooted tree is not itself a graph cycle.
// Target: computer-science.bigb
Dynamic programming Created 2026-09-24 Updated 2026-10-13
Dynamic programming solves a mathematical optimization problem by choosing reusable subproblems, recording their value functions, and combining already solved cases through a recurrence relation. Its attraction is that many possible decision histories can share one continuation calculation. A dynamic programming state must retain everything needed to determine the allowed future choices and their rewards. The 0-1 knapsack problem, for example, needs both the available capacity and which items remain available; capacity alone would lose the restriction that each item can be used at most once.
Two ideas explain when this works. Optimal substructure makes an optimal continuation valid inside a larger optimal solution. When subproblems overlap, storing each answer avoids repeated work: memoization computes entries as needed, while a table fills them in dependency order. Start from known base cases. For a problem indexed by increasing size, compute smaller cases first; for a finite sequence of future decisions, work backwards from the terminal reward, as in the Bellman equation. See Bertsekas, MIT lecture 2: optimality and state augmentation.
Consider a 0-1 knapsack problem with item weights , item values , and capacity . Define the value function to be the largest total item value obtainable from the first items with total weight at most . Here and specify the dynamic programming state. All weights and capacities are nonnegative integers. The empty selection supplies the base case . The recurrence relation is
The two candidates skip or take item . Both use the previous row, so taking item cannot take that same item again. The condition prevents a negative remaining capacity; no entry with is needed. The optimal substructure is concrete: after taking item , its companions must solve the smaller 0-1 knapsack problem on the first items with capacity . Otherwise replacing them by a better feasible selection would improve the whole selection. Compare [ref]; those notes index the remaining items in the opposite direction.
The full computation is small enough to see at once:
For example, and . Choosing the feasible item with largest individual value takes weight , earns , and leaves capacity , which cannot fit another item. This greedy algorithm misses the two-item selection of weights and , whose value is . This example tests the largest-individual-value rule; choosing by value divided by weight happens to find the optimum here.
Figure 1.
A knapsack table builds the optimal value from smaller subproblems
. The value function of this 0-1 knapsack problem is computed row by row. Outlined entries show the reconstruction of an optimal selection: skip weight 4, take weight 3, then take weight 2. The total weight is 5 and total value is 7. Details.
The number is the optimal value, while the choices attaining it form a control policy. To reconstruct one optimal selection, start at . Equality with allows skipping item . At , taking item gives , so continue to . Take item and finish at . In a tie, either maximizing choice suffices to recover one optimum. Recording choices during the calculation or comparing stored entries afterwards recovers the control policy; a final optimal value alone does not describe the selection.
Figure 2.
Knapsack dependencies and one optimal reconstruction
. Each node is a dynamic programming state labeled by its value function. Arrows point to the earlier-row entries used in the recurrence relation, forming a Directed acyclic graph. Dashed arrows skip an item; blue arrows take it. The orange graph path reconstructs the optimal selection with value 7. Compute in the opposite direction to the arrows, beginning at the zero base row. Details.
For items and integer capacity , the table uses time complexity and space complexity , counting arithmetic operations and stored numbers using Big O notation. This is pseudo-polynomial time: when weights and capacity use the binary numeral system, the capacity needs only digits. A capacity near creates about columns despite needing only about digits. Thus this algorithm is practical for moderate capacities but does not establish polynomial time in the encoded input length. See MIT 6.006 recitation 19: knapsack and input size. For the optimal value alone, one array reduces space complexity to : for each item, update capacities in descending order, from down to . Ascending order would read entries already updated for that item and could reuse it. Reconstruction then requires extra information or recomputation.
The link between the knapsack problem and dynamic programming is already explicit in a 1957 paper by Richard Bellman at RAND: its abstract discusses an exact knapsack problem solution using functional equations and cites his 1957 book. See Bellman, Comment on Dantzig's Paper on Discrete Variable Extremum Problems (1957).
Insertion sort 2026-10-13
Insertion sort grows a sorted prefix of an array one entry at a time. In the adjacent-swap version, start with position , compare its current sort key with the preceding sort key, and swap the complete sorting records only when the right sort key is strictly smaller. Continue leftward until no swap is needed or the entry reaches position zero; then process original position . A tie stops the pass. This is the strict comparison and exchange rule in the authored implementation.[ref]
Take , where the subscript labels a complete sorting record and comparisons inspect only its sort key. Starting from index zero, the complete key-comparison trace is:
ComparisonRight sort key smaller than left?ActionArray after the action
1, insert position : yesSwap positions .
2, insert position : noStop this pass on a tie.
3, insert position : yesSwap positions .
4, continue leftward: yesSwap positions .
5, continue leftward: noStop this pass on a tie.
Figure 1.
Insertion sort: five sort key comparisons and three complete-entry swaps
. Each row after the input shows the state after one comparison. Orange cells mark an adjacent swap; green cells mark a stopping tie. Identity letters are not compared. Both equal-key groups retain their input order, demonstrating stable sorting. Details.
The loop invariant is that, before pass , positions satisfy sortedness and contain precisely the entries originally in that prefix. Moving the next entry left through strictly larger sort keys extends that prefix, and every swap retains sorting permutation preservation. Equal-key entries never cross, giving stable sorting. Each leftward step decreases the position, and each completed pass increases the prefix length, so the algorithm terminates.
With constant-cost key comparisons and swaps, reverse-ordered distinct sort keys cause comparisons and swaps. Thus worst-case time complexity is in Big O notation. An already sorted array needs only key comparisons for and no swaps. The adjacent-swap version uses constant auxiliary space complexity. Small or nearly sorted arrays can therefore make insertion sort useful despite its quadratic worst case.
Longest increasing subsequence Created 2026-09-28 Updated 2026-10-14
A longest increasing subsequence is an increasing subsequence of maximum length. Write that length as , with for the empty sequence; a nonempty input has . Several selections can attain the same maximum. Here uses the weak convention ; write for the maximum length of a strictly increasing subsequence. For , these lengths are three and one respectively; for an input with distinct values they agree.
For , the answer is , attained at indices , while . One useful interpretation is the smallest number of deletions needed to make the surviving sequence nondecreasing:
Indeed, keeping a longest increasing subsequence attains this bound, and any valid remaining sequence is an increasing subsequence, so cannot have more than terms. In this example the answer is . This makes a measure of how much order can be recovered without moving any observation.
A table that ends at each position. Let be the maximum length of an increasing subsequence ending specifically at position . The dynamic programming recurrence relation is
The zero covers a selection consisting only of . Otherwise its penultimate index is an eligible ; append to an optimal selection ending there. Fill entries in increasing index order. For strictly increasing subsequences, change the comparison to . The strict version and reconstruction are developed in University of Toronto, CSC373 tutorial 4 solutions (2018), exercise 2; the weak version follows by allowing equality at the append step.
In the worked input, the weak table is and the strict table is . At the second , equality extends the first in the weak table. The final entry is only one, although the optimum is four: the best increasing subsequence need not finish at the last input position. Record a maximizing predecessor whenever . Following these stored indices from any position maximizing , then reversing them, gives an actual longest increasing subsequence. For the weak example the backward chain is .
Figure 1.
Skipping positions and allowing equal values changes the optimum
. An original toy input, not measured data. The colored selection keeps indices and values . The two dynamic programming rows give the best lengths ending at every position; the backward arrows reconstruct the weak longest increasing subsequence. Details.
Scanning all earlier positions for each uses time complexity and space complexity for lengths and predecessors, using Big O notation and assuming constant-time comparisons and operations on indices.
Keep the easiest ending to extend. The faster algorithm maintains a minimum-tail table for increasing subsequences: is the smallest final value of any length- increasing subsequence in the processed prefix. For equal lengths, a smaller ending permits every future extension allowed by a larger ending. Thus only one best ending per length is needed. The table is nondecreasing, since removing the last term of a length- selection gives . Start with an empty table. On reading , use binary search to find the first entry strictly greater than : replace it by , or append if there is no such entry. Its position is one beyond the longest existing length whose ending is at most . See Liben-Nowell, Vee and Zhu, section 3.1 and Figure 1.
Duplicates determine the binary search rule: weak increase replaces the first , whereas strict increase replaces the first . In Python these positions are returned by bisect_right(T, v) and bisect_left(T, v) respectively, counting positions from zero. Replace an entry or append; do not insert and shift the remaining entries. The documented equality rules are in Python documentation, bisect_left and bisect_right. Each search takes comparisons, giving time complexity , conventionally for , and space complexity for the weak length alone.
Figure 2.
Two binary-search tie rules give different minimum-tail tables
. Original exact traces for the same toy input, not measured data. Each cell shows a tail value and the input index supplying it; the outlined cell was updated in that row. At the second , the weak algorithm appends and the strict algorithm replaces. The final weak table has length four, but its supplier indices are not in increasing order. Details.
A length certificate needs separate reconstruction. The final weak tail values are , but they do not form an increasing subsequence of this input: the zero occurs last. Each table entry certifies a selection of its own length; the entries need not come from one compatible selection. To recover a longest increasing subsequence during the fast algorithm, store a supplier index for each tail and a predecessor index for each input position. Before replacing slot , save , or no predecessor if , then set . Later replacements leave saved unchanged. Follow predecessors from the final and reverse. This uses space complexity and returns indices for the weak trace. The same construction works with the strict comparator; compare the two auxiliary arrays in CP Algorithms contributors, restoring the subsequence after binary search.
Deleting or changing one term. If deletes position , then
Deleting cannot create a longer increasing subsequence; removing position from a previously optimal selection loses at most one term. If instead changes just the value at position , both inputs have the same deleted sequence, so their lengths each lie between its length and its length plus one. Consequently . These bounds also hold for strictly increasing subsequences. In the weak example, deleting either lowers the length to three, while deleting the final zero leaves it four. This is a Lipschitz bound for single-term edits, regardless of the numerical size of a replacement. Insertion has the same bound by reversing deletion, so insertions, deletions or replacements can change by at most .
For historical context, [ref] studies the comparison count for distinct elements of a totally ordered set. Its distinctness assumption makes the weak and strict conventions coincide. No distinctness assumption is needed for the comparator-specific algorithms above.
Under constant-cost lookup and integer addition, Memoized Fibonacci evaluation has time complexity in Big O notation, retaining values. A hash table needs its own lookup-cost assumptions; an array indexed by avoids hash collisions in this example. Memoization removes repeated bodies, but also pays for keys, lookups and retained answers. A workload without reuse can pay those costs without saving evaluations.
The call stack remains. Left-first recursion descends through arguments before returning, giving simultaneously active frames for . Therefore the auxiliary space complexity includes frames as well as table entries in the constant-size-value model. An iterative bottom-up dynamic programming calculation removes these recursive frames; retaining only the two previous Fibonacci numbers suffices when only the final value is wanted, but discards the table useful for later requests.
Exact Fibonacci numbers do not remain constant-size values. For , the recurrence relation gives , so the representation of as a binary number has order bits. With ordinary addition costing order bit operations on such operands, the additions cost order bit operations in total. Retaining all values also requires order bits of value storage, before keys and frame overhead. The linear bound counts additions under its stated cost model; it is not a linear bound on exact integer arithmetic in bits.[ref]
Merge sort 2026-10-13
Merge sort is a divide-and-conquer algorithm that sorts a finite array by splitting it into contiguous halves, sorting each half using recursion, and combining the two sorted arrays with the merge algorithm. An empty or one-entry array is returned immediately. For length , put the first entries on the left and the remaining on the right; the brackets are the floor function and ceiling function. Always choose the left head on equal keys to obtain stable sorting. This is the split, sort, combine structure taught in the MIT lecture notes.[ref]
Take the concrete input array
The tags identify the entries originally at indices and , counting from zero. Both keys are ; the tags are never compared. Reading downward in the figure follows the splits, then the returned sorted arrays. Arrows show dependencies; the level layout groups independent calls rather than giving their chronological execution order.
Figure 1.
Merge sort splits into base cases and returns sorted arrays
. The tagged keys 2a and 2b retain their original order through stable sorting. The merge algorithm makes 4, 6 and 7 head comparisons on the three returning levels, for 17 in total. Details.
The first two halves are and . Their pairs split into one-entry base cases. Applying the merge algorithm gives the following fully worked return levels; each number in the last column counts only comparisons between two available heads.
Returning levelResulting sorted arraysHead comparisons
Pairs, , ,
Halves,
Whole array
In the left-half merge algorithm, the head pairs are , and ; it takes , then appends . In the final merge algorithm, the head pairs are , , , , , and . It takes , then appends . Thus this precise version makes head comparisons and, ignoring identity tags, returns .
The elegance of merge sort is that it creates exactly the structure its next step needs: two smaller sorted arrays make the larger ordering inexpensive. Once the whole array is sorted, equal keys are adjacent and binary search can serve repeated queries. Stable sorting also retains information that the keys alone do not express. If the left-half merge algorithm instead takes the right head on the tie, it returns ; the keys are sorted but stable sorting has failed.
For time complexity, assume each key comparison, entry access and entry copy has constant cost. The merge algorithm processes entries in time. In Big O notation, the resulting recurrence relation is
For , the balanced levels contain entries each and there are returning levels. Each costs , so the worst-case time complexity is . For arbitrary , there are at most returning levels, each processing at most entries, giving the same bound. The worst-case number of head comparisons is also . Here is the base-two logarithm. The MIT notes give this level-by-level analysis for merge sort.[ref]
The conventional array implementation uses a reusable buffer of entries for the merge algorithm, together with pending calls for recursion. Its auxiliary space complexity is therefore , beyond the original array. This bound describes that implementation. A linked list permits a merge algorithm that reconnects existing nodes; its storage accounting differs from copying array entries into a buffer.
For a short algorithm correctness outline, the empty and one-entry base cases are sorted. Assuming recursion correctly sorts both shorter halves, the merge algorithm returns all their entries in order. Equal keys within a half keep their relative order by the same argument applied to the smaller call; across halves, every left entry originally preceded every right entry, so taking left first preserves stable sorting. Both half-lengths decrease for , establishing termination.
Tridiagonal solve costs 2026-10-14
For the displayed unpivoted Thomas algorithm with one right-hand side, count each scalar addition, subtraction, multiplication or division as one arithmetic operation. Forming takes five arithmetic operations per forward step. The last unknown takes one division; each remaining back substitution step takes three arithmetic operations. Thus
This is an operation-count model, excluding assignments, indexing, pivot checks and any special simplifications of the coefficients. It is not a timing benchmark. The original tridiagonal matrix needs scalar entries, and its right-hand side another . Overwriting with and with , then with , keeps storage ; the arrays, rather than the few loop temporaries, determine the total space complexity.
For repeated right-hand sides, retain the LU decomposition: computing once costs arithmetic operations, and each subsequent pair of forward substitution and back substitution sweeps costs . Hence right-hand sides cost arithmetic operations. This is the banded counterpart of Reusing LU factors for multiple right-hand sides.