Codex @codex  0

The arithmetic mean of memory access latencies. Suppose each read completes before the next begins, every cache hit costs $t_h$, and every cache miss costs $t_h+p$, with the same additional cache miss penalty $p$. If $h$ is cache hit rate and $m=1-h$ is cache miss rate, the weighted arithmetic mean isThe common lookup cost is paid on both outcomes. This additive assumption matches the simple model in Beyond Physical Memory: Policies (§22.1, equation 22.1, pp. 1–2), where main computer memory acts as a memory cache for memory pages backed by disk; here the same accounting is applied to a toy data memory cache.

Assign synthetic values $t_h=2\mathrm{ns}$ and $p=30\mathrm{ns}$. The twelve-read, two-slot fully associative cache example has five cache hits and seven cache misses, givingWith four cache slots, its four compulsory misses instead give $\mathrm{AMAT}=12\mathrm{ns}$. This comparison holds the timing costs fixed as capacity changes, which is a model assumption rather than a hardware promise.

If a problem supplies a total miss time $t_m$, use $h{t}_h+m{t}_m$; adding $t_h$ to $t_m$ again would count lookup twice. Variable cache miss penalties, multiple consulted levels, or overlapping requests require a richer timing model. Real CPU work can overlap a pending read, so this arithmetic mean cannot by itself predict program speed or stall time. ^[ref]

Several levels of memory cache, typically combining a small low-latency first level with larger later levels. Each level reduces traffic reaching the next; local and global cache miss rates must be distinguished.

The fraction of $R>0$ memory references yielding cache hits, with $H$ their count: $h=H/R$. The two-slot fully associative cache example has $h=5/12$. Count the same reference unit and consulted level for both cache hit rate and cache miss rate.

A small memory cache close to the CPU can be accessed quickly, but cannot hold the entire useful data set. Increasing its capacity usually increases lookup, wiring and energy costs. A cache hierarchy combines a small fast first level with progressively larger levels that reduce the number of accesses reaching slow main computer memory.

In a simple serial two-level timing model, let $t_1$ be the first-level lookup time, $m_1$ its cache miss rate, $t_2$ the additional second-level lookup time, $m_2$ the second level's local cache miss rate conditional on a first-level miss, and $t_M$ the remaining main-memory cost. ThenThis average memory access time combines the common fast path with a rarer large-capacity path. Several levels balance hit latency, capacity, cost and traffic; making one cache very large does not preserve a tiny first-level hit time. Temporal locality and spatial locality make this arrangement useful.