Evolutionarily stable strategy ($e(x^{\ast },(1-ϵ)x^{\ast }+ϵy)>e(y,(1-ϵ)x^{\ast }+ϵy)$)

A resident strategy is evolutionarily stable if every distinct mutant earns lower expected fitness when introduced at sufficiently small positive frequency. The upper bound on the frequency may depend on the mutant. In a finite symmetric matrix game, $e(x,y)=x^{T}Ay$ is the expected payoff, with the second argument representing the population strategy. Affineness in this argument yields the two-condition criterion for evolutionary stability. Every evolutionarily stable strategy is a symmetric Nash equilibrium, but a tied best response must also be strictly outperformed against its own mutant population. The Hawk-Dove game supplies an interior stable mixture despite equality of all payoffs against the resident mixture.

For a differentiable rare-mutant invasion fitness $f(y,x)$ against resident trait $x$, the selection gradient gives the first-order growth advantage of a nearby mutant. A zero gradient is a candidate interior evolutionary singularity. Strict local uninvadability follows from negative mutant-fitness curvature; evolutionary approach additionally requires the gradient to point toward the candidate. Boundary traits require one-sided tests. A gradient alone does not prove global evolutionary stability.

Hawk fights for a resource of value $V$, while Dove retreats from Hawk and shares with another Dove. Hawk–Hawk competition incurs damage cost $D$. The expected payoff difference between Hawk probabilities $p,r$ against population probability $q$ is $(p-r)(V-Dq)/2$. For positive $V,D$, pure Hawk is an evolutionarily stable strategy when $V>D$. At $V=D$, it remains stable because a mutant $p<1$ loses $V(1-p{)}^2/2$ against its own population. When $0<V<D$, the stable mixture has Hawk probability $p^{\ast }=V/D$: all first-condition payoffs tie, and its second-condition advantage is $D(p-p^{\ast }{)}^2/2>0$. For nonpositive resource values, pure Dove is stable if $V<0$ or if $V=0<D$. At $V=D=0$ all payoffs vanish and there is no evolutionarily stable strategy.

The Chicken payoff ordering describes a symmetric two-player bimatrix game in which aggression earns temptation $T$ against yielding, mutual yielding earns $R$, yielding to aggression earns $S$, and mutual aggression earns $P$, with the displayed strict ordering. Aggression is a best response to yielding, and yielding is a best response to aggression. Thus the two asymmetric action pairs are pure strategy Nash equilibria. The Hawk-Dove game has this ordering in its positive-resource regime where the damage cost exceeds the resource value. In the prisoner's dilemma, the ordering of $S$ and $P$ is reversed, making defection a strictly dominant strategy instead.

For mixed-strategy expected payoffs put $A_y=e(x^{\ast },x^{\ast })-e(y,x^{\ast })$ and $B_y=e(x^{\ast },y)-e(y,y)$. The invasion payoff difference is exactly $(1-ϵ)A_y+ϵB_y$. Positivity for all sufficiently small positive $ϵ$ forces $A_y\ge 0$ by taking the limit. When $A_y=0$, it forces $B_y>0$. Conversely $A_y>0$ makes the difference positive for sufficiently small frequency, and $A_y=0,B_y>0$ makes it positive for every positive frequency. Apply this separately to every distinct mutant. This proves the equivalence with an evolutionarily stable strategy; affineness in population composition is essential to the formula.