For the symmetric zero-diagonal weights and asynchronous rule just specified, update only unit . In the energy difference, terms involving neither nor its symmetric partner cancel. Thus asynchronous Hopfield energy descent gives
If the unit was already aligned, it does not change and . If it flips into alignment, , so
At our tie rule retains the old activity. The network has only configurations. Strict descent on every actual change prevents revisiting a configuration and permits only finitely many changes. With a fair schedule every misaligned unit is eventually updated, so the terminal configuration is a fixed point. At that point each possible single-bit flip has energy cost . The network reaches a minimum against single-unit changes, not necessarily the global minimum. Its energy is a Lyapunov function for these asynchronous dynamics.
The qualifications are essential to the printed claim. Synchronous updates can cycle: for two units with , the state alternates with , keeping energy . Symmetry alone therefore does not prove convergence under simultaneous updates.
Nor does the valid descent proof guarantee a global minimum. Consider four units with and all four cross-group weights equal to , symmetrically and with zero diagonal. The state has aligned fields and energy , so it is a strict single-flip local minimum. The all-positive state has energy . This explicit counterexample shows why local minima may trap retrieval even though the energy always descends under the stipulated rule.
A standard binary Hopfield network contains recurrently connected units with activities . Choose symmetric weights , omit self-couplings by setting , and use zero thresholds for the energy in the next part. To store desired binary patterns , a simple choice is Hebbian memory weights for a Hopfield network:
Correlated activity strengthens connections between units agreeing across the training patterns and makes disagreement contribute negatively. This creates attraction toward the stored configurations when cross-talk is sufficiently small. For one stored pattern, its local field is , so every bit is stable for . With multiple patterns, the other patterns add cross-talk; the rule does not guarantee exact retrieval for an arbitrary collection.
Start retrieval from a partial or corrupted cue and update one unit at a time using its current field
A cyclic or random fair asynchronous schedule lets the network relax toward a fixed point, completing a memory when the cue lies in its basin. This is content-addressable retrieval: the initial pattern selects the memory without an explicit address. Nonzero thresholds can be included in a more general model, but their corresponding linear terms must then be included in the energy.
The Jacobian matrix at a diagonal steady state is
Its eigenvectors and represent common and opposing perturbations, with respective eigenvalues and .
For a nonzero equilibrium, and . Consequently . The lower root has and is a saddle equilibrium; the upper root has and is a stable node. At the origin , so both eigenvalues are .
The supplied parameters give , and the nonzero roots solve , giving and . The numerical stability results are
Biologically, the origin is a quiescent state and the upper equilibrium a sustained mutually reinforced activity state. The saddle represents a collective activation threshold: a perturbation crossing its basin boundary can switch the network from low to high activity. This persistent-state mechanism can model a memory or decision variable, but the simplified rate equations do not establish that a particular biological circuit uses it.
Put . The nullclines are
Any steady state has nonnegative coordinates because . For , is strictly increasing on the positive half-line. If a steady state had , then , a contradiction. The case is identical, so all steady states lie on the diagonal. Write . Besides , the equation gives
and hence
Thus three distinct steady states exist exactly when , with and the positive parameter assumptions stated above. For positive excitatory coupling this is . At equality the two positive states merge; below it only the origin remains. If , the origin is again the only steady state.
Both nullclines begin at the origin and approach the limiting activity along their saturating direction; they are reflections of each other across . Flow points to the right where and to the left where . It points upwards where and downwards where . The plot shows these nullclines and typical trajectories for the parameters of the next part.
Figure 1.
Nullclines and flow of the bistable two-neuron rate network
. Details.
The origin and upper diagonal state attract nearby trajectories, while the lower positive state is a saddle. Its stable manifold separates the two attracting regions: trajectories in each basin of attraction approach its stable node, giving bistability. The diagonal is invariant, and on it the scalar activity increases between and and decreases between zero and or above . Activities remain nonnegative and bounded because . The vector field has constant negative divergence , so the Bendixson-Dulac criterion also rules out periodic orbits in the physical quadrant. The network is a switch between persistent activity levels, rather than an oscillator.
Interpret and as nonnegative firing-rate activities, rather than individual action potentials. Take . The relaxation time controls how quickly activity changes. The decay terms and make activity fall in the absence of recurrent drive. Each neuron receives a weighted input from the other, with coupling , and the static input-output function converts that input into a target activity.
The nonlinear response tends to at large input magnitude and equals at , so is its maximum activity and its half-saturation input scale. At small inputs it is approximately . For this is a mutually excitatory saturating rate network: increasing one neuron's activity increases the other's drive, permitting positive feedback. The response is even in , so this particular equation does not model a negative as ordinary inhibition; its dynamics depend on .
The equations combine passive relaxation with instantaneous nonlinear recurrent drive. They omit explicit spike timing, delays, channel dynamics and stochasticity, so they are an activity-level model for computational neuroscience.
Assume the two response labels are distinct and the error probability is the same for each stimulus. Then the stimulus-response map is a binary symmetric channel:
with the two incorrect conditional probabilities equal to . Equal stimulus probabilities give . In bits, the Shannon entropy of the response is therefore . For either stimulus the response entropy is the binary entropy
with . Thus , and the mutual information is
It is maximal, equal to one bit, at and also at . The latter response is always opposite to the ideal label but still specifies the stimulus perfectly after relabelling. It is minimal, equal to zero, at , when both response distributions are identical and the response is independent of the stimulus. If error is conventionally restricted to , the maximum occurs only at zero.
The symmetric-error assumption matters. A single overall error probability would not determine this information if the two stimuli had different conditional error rates or if the response included additional outcomes.
One useful starting point is reverse correlation, implemented as a spike-triggered average. Present a long stimulus record and retain the stimulus history before each observed spike. After subtracting the stimulus mean, estimate
where is the number of spikes and is the history window. This identifies a candidate temporal receptive field. For white Gaussian stimulation and a one-filter encoding model, the average is proportional to the relevant filter, provided the response nonlinearity does not make that average vanish. For example an even response to the projection can have zero average despite strong stimulus dependence. Colored inputs require accounting for stimulus covariance, and more than one relevant stimulus direction may require a richer fitted model.
Use the candidate filter in an LNP model:
Estimate the static nonlinearity by binning filter projections and dividing spike counts by exposure time in each bin. Equivalently, Bayes' rule gives under this model. Refine the filter and nonlinearity by maximizing the conditional Poisson log-likelihood
with suitable regularization. Simulate an Inhomogeneous Poisson process at the fitted rate using integrated-intensity inversion from the previous solution. This gives a statistical encoding model, not a claim to reconstruct the neuron's ion channels.
Use cross-validation: estimate parameters on one set of trials and test them on independent held-out trials. Under repeated identical stimuli, compare the observed and simulated trial-averaged firing rates, spike-count distributions, interspike-interval distributions and autocorrelations. Compare held-out likelihoods and predictive errors with simpler baseline models and with the variability between real repeats. Exact spike-by-spike agreement is not expected from two independent realizations of a stochastic model. Large rate agreement accompanied by incorrect short-interval or correlation statistics reveals missing dynamics. Adding a spike-history term, for example in a point-process generalized linear model, can account for refractoriness or bursting when the basic Poisson cascade fails.
A good model predicts held-out response distributions, not just its training spike train. Such a model summarizes what stimulus features affect the neuron, permits controlled simulated changes of those features, and tests whether the proposed computation is sufficient. Its failures can suggest further experiments or additional stimulus and history variables.
For a constant-rate Poisson process, let be the waiting time from a spike to the next spike. By independent increments, there is no spike in an interval of length with probability . Hence, for ,
Its density is , so the interspike intervals are independent exponential distributions with mean . If is uniform on , inverse transform sampling gives . Repeatedly add independent draws to the current spike time to simulate a constant-rate spike train.
For a prescribed time-dependent nonnegative rate , use an Inhomogeneous Poisson process. Conditional on the present event at , its next waiting time has survival probability
Thus its conditional CDF is minus that expression for . Define the cumulative intensity . For each new independent uniform draw, set and solve
This transforms independent unit-rate exponential intervals in integrated-intensity time into physical spike times. Use a generalized inverse when the rate vanishes on an interval: no spikes are placed in a flat part of . On a finite simulation window, stop when the next intensity target exceeds the available . If the total future intensity is finite, a target beyond it means that no further spike occurs.
Numerically, integrate the known rate and invert the accumulated integral, or integrate exactly across piecewise-constant rate bins. Drawing an exponential interval using only is not exact when the rate changes during the interval. The Poisson model also omits the neuronal refractory period and other spike-history effects.
An action potential is a brief regenerative change in a neuron's membrane potential, followed by repolarization and recovery. In a typical spiking neuron, depolarization recruits voltage-gated ion channels; inward current promotes further depolarization until inactivation and opposing currents terminate the event. The neuronal refractory period makes immediate repetition difficult or impossible. Once initiated, the roughly stereotyped event propagates along the axon. Its amplitude is therefore not generally the primary variable used to represent stimulus strength.
Inputs arriving at synapses on a neuron and its dendrites change membrane currents and produce graded voltage changes. Their spatial and temporal integration can trigger an action potential. At a chemical synapse, an arriving spike promotes release of a neurotransmitter, whose receptors alter the next neuron's conductances. Depending on the synapse and receptor mechanism, the input can promote or suppress the next neuron's firing. Thus information flows through activity patterns interacting with connectivity and synaptic dynamics, rather than by an identical voltage waveform simply passing unchanged across every cell.
In rate coding, a relevant response statistic is the number of spikes in a chosen interval, or the mean firing rate inferred from repeated trials or a population. For example, a sensory variable might be represented by mean counts of two versus five spikes in a ms window. A downstream neuron integrating many synaptic inputs over that window can distinguish the rates even if individual spike times vary. The window, averaging method and neuron's stimulus tuning must be specified; not every cell increases its rate for every stronger stimulus. Neural adaptation can also make the rate depend on recent stimulus history.
In temporal coding, precise times carry information beyond that count. Two responses containing the same number of spikes can differ in their first-spike latency after stimulus onset. Alternatively, spikes can lock to a particular phase of a periodic input, or relative timing across two neurons can encode coincidence. A decoder that reads only the long-window count discards those distinctions, whereas a temporally sensitive synapse or coincidence detector can use them.
Rate coding uses a chosen frequency or count summary; temporal coding uses the arrangement of spike times relative to a reference. They are complementary possibilities within neural coding, and a spike train can carry information in both. Stochastic variability does not prevent coding, but requires assessing how reliably different response distributions can be distinguished.
Choose linearly independent basis functions in the zero-trace Sobolev space, and put . A concrete conforming Galerkin method uses continuous piecewise-linear hat functions on an interval mesh, with zero endpoint values. Seek satisfying
Testing against each basis function gives
The matrix is symmetric, and for a nonzero coefficient vector , its represented function is nonzero by basis independence. The coercivity estimate gives . Thus is a positive-definite matrix, the finite-dimensional problem has a unique solution, and it minimizes the same energy over . This is Ritz-Galerkin equivalence for a symmetric coercive form.
Subtracting the continuous and discrete equations proves Galerkin orthogonality: for all . Let . For any comparison , orthogonality and the Cauchy-Schwarz inequality in this inner product give
Therefore
With , the same argument in the derivative norm gives the Céa lemma estimate .
Convergence now follows from approximation, not from a bare assertion that the residual is small. The defining density of smooth compactly supported functions in lets one approximate by such a smooth function. Its piecewise-linear interpolant has zero endpoints and derivative error tending to zero as the largest mesh interval tends to zero. For a smooth , the derivative on each element is the average of there, and the interval Poincaré inequality bounds its error by the element length times . Density and the boxed best-approximation estimate consequently give in , and hence in , without assuming a bounded derivative of . If the stronger regularity is available, the same interpolation estimate gives first-order convergence in the derivative norm.
Work with real functions and . To give the variational problem a definite domain, choose the homogeneous Dirichlet boundary conditions used in the next part. The natural form space is the zero-trace Sobolev space , with norm . The Sobolev trace operator is well defined because one-dimensional functions have absolutely continuous representatives. For example , so its endpoint trace is bounded in the norm; the other endpoint has the analogous estimate. The zero-trace space is closed in the Hilbert space , and the Poincaré inequality makes the chosen derivative norm equivalent to its full norm. Thus is a Hilbert space.
Differentiability of implies continuity on the compact interval. Hence there are finite constants and a positive with and . Define the symmetric bilinear form
The Cauchy-Schwarz inequality and Poincaré inequality give all the estimates required by the Lax-Milgram theorem:
That theorem states that a continuous coercive bilinear form on a Hilbert space and a bounded linear functional determine a unique weak solution. Completeness, continuity, coercivity and boundedness have now all been verified. Therefore there is a unique with for every .
The energy version of the given variational problem is . Its exact variation is
Boundedness of makes the last term quadratic in , so the Fréchet derivative is . Its vanishing for every test function is precisely
Taking compactly supported smooth tests and integrating in the distributional sense shows that this is the Euler-Lagrange equation . Conversely that weak equation gives the displayed stationary condition. The stationary point is actually the unique global minimizer, because
Equality occurs only when . This proves the variational equivalence, including existence and uniqueness rather than only a formal derivative calculation.
For precision about the literal L2 inner product , use the flux-domain realization of a one-dimensional elliptic operator
For such , weak integration by parts and its zero endpoint values give . On the larger form space , the same energy is interpreted by the weak operator pairing. The minimizer belongs to automatically: its weak equation gives , while . Thus its equation also holds as an equality of functions. This domain does not assume bounded , which mere differentiability would not supply. If one may equivalently use , but that stronger coefficient assumption is not needed for the proof here.
Take interior points in each direction and spacing . The one-dimensional Dirichlet discrete Laplacian has sine eigenvectors , because
The five-point Dirichlet Laplacian as a Kronecker sum therefore has product eigenvectors . Adding the reaction term gives modal growth rates
The semidiscrete matrix is real symmetric, so these modes are orthogonal in the mesh-weighted discrete L2 norm. Non-growth for all times is equivalent to every rate being nonpositive. The largest is , giving the five-point heat-reaction stability threshold
At equality the first grid mode is neutral; above it that mode grows exponentially. This criterion concerns the spatially semidiscrete, continuous-time evolution. Any subsequent time integrator has an additional stability requirement.
The threshold expands as and is below the continuum threshold for every finite mesh. Thus at the continuum neutral value the grid first mode has a small positive rate. This is compatible with finite-time convergence, although it prevents an all-time contraction on that fixed mesh. In the weaker finite-time mesh-uniform sense, all fixed have the bound because the discrete Laplacian is nonpositive. The boxed range is the natural non-growing stability range, matching the threshold comparison in the preceding part; finite-time stability versus power boundedness is a distinct question.
The Fourier sine basis of the unit square is , , normalized in the L2 inner product. It satisfies the zero Dirichlet boundary conditions and . Expand the given initial datum as , where . The solution is
For , every exponent is nonpositive. The series converges in for , tends to in as by dominated convergence, and differentiates termwise for because the Gaussian high-frequency decay dominates every polynomial derivative factor. This constructs a classical positive-time solution and the appropriate mild solution at the possibly rough initial time.
Its norm and the difference of two such solutions satisfy
Existence, uniqueness in the natural solution class, and continuous dependence follow. For uniqueness one can project any weak solution onto each sine mode, obtaining the same scalar linear equation and initial coefficient. Thus this is a Hadamard well-posed problem, with a contractive strongly continuous semigroup in the stated range.
To prove pointwise decay from merely data, use and the Cauchy-Schwarz inequality:
For , factor out ; the remaining sum is bounded by . Therefore
At equality the first sine mode is stationary, so strict inequality is essential for decay of every datum. This is the Dirichlet heat-reaction threshold on the unit square. The equation is actually well posed on every fixed finite time interval for any fixed real ; beyond the threshold it has a growing first mode and loses long-time contractivity. The restricted well-posedness assertion in the question is therefore sufficient rather than necessary.
For a Fourier mode , define
The amplification factor is . To derive its stability condition, use the coefficient abbreviations from the preceding solution and set . The squared moduli are
Here , , and . Subtraction and factorization give
Thus for every frequency exactly when , provided the implicit denominator does not vanish. For the physically nonnegative Courant number, this gives
For a signed convention the full real range is .
There is no hidden denominator failure in these ranges. If and , the identity forces and ; but . If , then , and could vanish only when its two real coefficients have equal magnitudes. Their sum is three, so equality requires , which is not a root of . Hence is nonzero for every frequency in the allowed ranges. On the compact frequency interval its inverse is bounded for each fixed allowed .
When , choose a small nonzero frequency. The denominator remains close to three, while the boxed identity gives . Its fixed-frequency numerical mode grows geometrically, proving instability in the usual fixed-Courant refinement limit. Conversely, for , the Plancherel theorem converts the mode bound into a discrete L2 norm contraction on the whole-line or periodic grid. The large-Courant branch is possible because the scheme is implicit; stability alone does not promise small error at a large step.
Figure 1.
Fourier amplification of the implicit advection stencil for Courant numbers 0.5, 1.5 and 2.5
. Details.
Let the spatial spacing be , the time step , and . Abbreviate the three new-time coefficients by , , , and the two old-time coefficients by , . These are the coefficients of the compact third-order advection stencil.
For a smooth solution of the advection equation, implies . Substituting the exact solution into the stencil therefore evaluates all five terms on the same characteristic profile, at spatial offsets , , , , and . Its Taylor moments are
Direct polynomial expansion gives
Consequently the unscaled local truncation error is
At a fixed nonzero Courant number, dividing by the time step gives an consistency defect. Thus
When , the order is higher because the method simplifies to an exact integer shift: at the two sides have the common factor and the amplification is ; at it is . The signed value gives the shift , and gives the identity. These special cases do not invalidate the third-order statement for the family. Under the stability conditions established next, the usual accumulation of the fourth-order local defect over steps gives third-order convergence for smooth data.
Define the stage states , so . Skew symmetry gives , and hence
Insert this into the preceding norm identity and symmetrize the double sum, using . The result is the Runge-Kutta conservation of quadratic invariants identity
Thus
which proves the sufficient condition for every well-defined step and then for all by induction. For example the implicit midpoint rule, with and , has and preserves this norm exactly. The condition is sufficient, and need not be necessary for a particular matrix or a particular solution.
The Runge-Kutta method has update . Expanding its squared Euclidean norm gives the exact identity
The stated per-step relation makes this difference zero. Induction from the initial vector then proves
For a positive step size the relation is also necessary for exact per-step norm conservation. This algebraic identity applies whenever the stage equations define a step; it does not silently assume solvability of arbitrary implicit stages.
For a real skew-symmetric matrix, . A scalar quadratic form satisfies , so it is zero. Therefore
The finite-dimensional linear system has the global solution . Equivalently, , so its exact flow is an orthogonal matrix and preserves the Euclidean norm.
Apply the method to the Dahlquist test equation , so , and put . A trial numerical mode satisfies
For every real , the leading coefficient is positive, while
The intermediate value theorem gives a real amplification root . Its modulus exceeds one although the exact scalar solution is decaying. Therefore
For a concrete choice, gives , whose negative root is . This proves negative-real instability of the symmetric two-derivative formula directly, without using either an order barrier or the supplied quadratic criterion. The unstable branch is the parasitic amplification root issuing from at ; its small-step expansion is , so a negative moves it outside the unit circle immediately.
Along a smooth exact solution, the chain rule gives . Expand the unscaled defect of the multiderivative multistep method about the central time . The antisymmetric difference of the solution and second derivative, and the symmetric sum of first derivatives, give
All derivatives are evaluated at . Reflection symmetry removes every even power. The fifth-order conditions require the coefficients of to vanish. The last two equations give and , and the first then gives . Thus
These determine the symmetric two-step two-derivative formula. Substitution into the next coefficient gives
The requested fifth-order conditions actually produce formal order six. In the usual maximal-order terminology there is no member of this symmetric family with exact order five: cancelling the fifth-power defect also cancels the automatically zero sixth-power defect. Interpreting the printed request as order at least five gives precisely the boxed parameters.
At zero step size the recurrence polynomial is , with simple roots and , so it satisfies the root condition for a multistep method and is zero-stable. With sufficiently smooth locally Lipschitz derivative maps, the nearby implicit solution branch and starting values of matching accuracy, the convergence of a zero-stable multiderivative method gives global order six. Formal order by itself does not imply decay of its additional numerical mode.

Pinned article: Introduction to the OurBigBook Project

Welcome to the OurBigBook Project! Our goal is to create the perfect publishing platform for STEM subjects, and get university-level students to write the best free STEM tutorials ever.
Everyone is welcome to create an account and play with the site: ourbigbook.com/go/register. We belive that students themselves can write amazing tutorials, but teachers are welcome too. You can write about anything you want, it doesn't have to be STEM or even educational. Silly test content is very welcome and you won't be penalized in any way. Just keep it legal!
We have two killer features:
  1. topics: topics group articles by different users with the same title, e.g. here is the topic for the "Fundamental Theorem of Calculus" ourbigbook.com/go/topic/fundamental-theorem-of-calculus
    Articles of different users are sorted by upvote within each article page. This feature is a bit like:
    • a Wikipedia where each user can have their own version of each article
    • a Q&A website like Stack Overflow, where multiple people can give their views on a given topic, and the best ones are sorted by upvote. Except you don't need to wait for someone to ask first, and any topic goes, no matter how narrow or broad
    This feature makes it possible for readers to find better explanations of any topic created by other writers. And it allows writers to create an explanation in a place that readers might actually find it.
    Figure 1.
    Screenshot of the "Derivative" topic page
    . View it live at: ourbigbook.com/go/topic/derivative
  2. local editing: you can store all your personal knowledge base content locally in a plaintext markup format that can be edited locally and published either:
    This way you can be sure that even if OurBigBook.com were to go down one day (which we have no plans to do as it is quite cheap to host!), your content will still be perfectly readable as a static site.
    Figure 5. . You can also edit articles on the Web editor without installing anything locally.
    Video 3.
    Edit locally and publish demo
    . Source. This shows editing OurBigBook Markup and publishing it using the Visual Studio Code extension.
  3. https://raw.githubusercontent.com/ourbigbook/ourbigbook-media/master/feature/x/hilbert-space-arrow.png
  4. Infinitely deep tables of contents:
    Figure 6.
    Dynamic article tree with infinitely deep table of contents
    .
    Descendant pages can also show up as toplevel e.g.: ourbigbook.com/cirosantilli/chordate-subclade
All our software is open source and hosted at: github.com/ourbigbook/ourbigbook
Further documentation can be found at: docs.ourbigbook.com
Feel free to reach our to us for any help or suggestions: docs.ourbigbook.com/#contact