 Solution

The payoff $e(x,z)$ is the expected fitness of an individual using strategy $x$ against a population using strategy $z$. The resident $x^{\ast }$ is initially common; the mutant $y$ is introduced at small frequency $ϵ$. Convexity of the strategy set makes their population mixture admissible. If residents have higher expected fitness in every sufficiently small invasion by a fixed mutant, selection works against that mutant: this is an evolutionarily stable strategy. The allowed upper bound on $ϵ$ may depend on $y$.

For expected payoffs of mixed strategies, $e(x,z)$ is affine in the population strategy $z$; in a finite matrix game it is $x^{T}Az$. This property is the basis of the two-condition criterion for evolutionary stability. For fixed $y\ne x^{\ast }$, putAffineness gives the exact invasion differenceIf this is positive for all sufficiently small $ϵ>0$, taking $ϵ\downarrow 0$ gives $A_y\ge 0$. If $A_y=0$, the difference is $ϵB_y$, so necessarily $B_y>0$. Conversely, if $A_y>0$, it remains positive for sufficiently small $ϵ$: for example take $ϵ<A_y/(A_y+|B_y|)$. If $A_y=0$ and $B_y>0$, it is positive for every $ϵ>0$. Applying this argument separately to every $y$ provesThus an evolutionarily stable strategy is a symmetric Nash equilibrium with an additional condition against tied mutants. For a general nonlinear interaction function on a convex set, affineness would be an additional necessary hypothesis for this particular equivalence; here it follows from the mixed-strategy expected-payoff interpretation. For example, on $X=[0,1]$ take $e(x,z)=xz(x/2-z)$ and resident $x^{\ast }=0$. For every $y>0$, $A_y=0$ and $B_y=y^3/2>0$, but at population $ϵy$ the mutant earns $ϵy^3(1/2-ϵ)>0$ while the resident earns zero. Thus the two conditions alone would not suffice for an arbitrary nonlinear $e$; the expected-payoff assumption is used explicitly.

For the Hawk-Dove game, let $p$ be an individual's Hawk probability and $q$ the population's Hawk probability. The given matrix yieldsWe use this formula in each case. In the usual interpretation, resource value $V$ and damage cost $D$ are positive. The parts below also specify the boundary or negative-value alternatives because positivity is not written explicitly in the PDF.