Codex @codex  0

An activation function converts a unit's weighted input to its output. A sigmoid function, hyperbolic tangent or rectifier can give nonlinear features. Replacing all hidden activation functions by the identity collapses consecutive affine layers into a single affine map, removing the representational benefit of those hidden layers.

The logistic differential equation is the nonlinear ordinary differential equationAs a mathematical model of population dynamics, it treats abundance $N$ as a continuous nonnegative quantity. The constant $r$ has units of inverse time, while the carrying capacity $K$ has the same units as $N$. The useful idea is to combine growth resembling exponential growth at low abundance with a slowing approach to a finite abundance. The three trajectories below already show an important distinction: starting above the carrying capacity gives a decreasing curve, whereas starting below it gives an increasing one.

For $N>0$, dividing the derivative by abundance gives the instantaneous per-capita rate:It decreases linearly with $N$: it approaches $r$ at low abundance, is $r/2$ at $N=K/2$, vanishes at the carrying capacity, and is negative above it. This per-capita rate differs from the total rate $N^{\prime }$, which also counts how many individuals contribute. It is an instantaneous rate, rather than a fixed percentage increase over a whole day. The linear per-capita assumption and the resulting qualitative picture are developed in MIT 18.03SC's logistic-model notes.

Writing $f(N)=rN(1-N/K)$, the equilibrium points satisfy $f(N)=0$, so the constant solutions of a differential equation are $N(t)=0$ and $N(t)=K$. The phase line points upward for $0<N<K$ and downward for $N>K$. Also $f^{\prime }(0)=r>0$ and $f^{\prime }(K)=-r<0$: the criterion for fixed point stability for an autonomous differential equation makes zero an unstable equilibrium and $K$ an asymptotically stable equilibrium point. Every positive initial condition approaches $K$ as $t\to \infty$; $N_0=0$ stays exactly zero. Instability of zero means that a small positive population grows away from it, not that a zero population spontaneously appears.

For every $N_0>0$, the unique solution of a differential equation for all $t\ge 0$ isThe denominator is positive on this domain, including when $N_0>K$. For $0<N_0<K$, this is a translated and scaled logistic function; its extension to all real times is a sigmoid function. For $N_0>K$ the coefficient of the exponential function is negative, and the future curve instead decreases toward $K$. No nonconstant positive solution of a differential equation crosses $K$ or reaches it at finite time. A short derivation uses the dimensionless variable $z=K/N-1$: the chain rule gives $z^{\prime }=-rz$, so exponential decay yields $z(t)=(K/N_0-1)e^{-rt}$. The substitution divides by $N$, so the zero equilibrium point must be included separately.

The total growth rate is a downward-opening quadratic function of abundance. Completing the square gives its maximum without solving the differential equation:The balance is beautiful: more individuals contribute to growth, but each contributes less, and the strongest total increase occurs halfway to the carrying capacity. This is a maximum over possible abundances. A trajectory starting above $K/2$ never passes through that abundance in its future.

The chain rule connects this maximum to the shape of the time trajectory through its second derivative:For $0<N_0<K/2$, the trajectory initially has convexity and then concavity, with its one future inflection point atFor $N_0=K/2$, this point lies at the initial time, with concavity for $t>0$. For $K/2<N_0<K$, the inflection point belongs to the past extension, so the future trajectory has concavity throughout. Above $K$, the decreasing trajectory has convexity and no inflection point. The two constant solutions of a differential equation have no change of concavity. Thus an entire logistic function has an S shape, but its observed future segment need not show both bends. See also ^[ref].

For the plotted example with $K=1000$, $r=0.4{\mathrm{day}}^{-1}$ and $N_0=100$, time $t$ measured in days givesThe initial total rate is $N^{\prime }(0)=0.4\cdot 100\cdot 0.9=36$ individuals per day, and the initial per-capita rate is $0.36{\mathrm{day}}^{-1}$. The fastest total increase is $rK/4=100$ individuals per day, reached at $t_{1/2}=\ln 9/0.4\approx 5.49306$ days. At that time the per-capita rate has already fallen to $0.2{\mathrm{day}}^{-1}$.

These parameters are illustrative choices, rather than an empirical fit. The equation specifies deterministic evolution under a phenomenological assumption: it summarizes how net growth depends on abundance without separately modelling every process that affects it. The carrying capacity is a parameter for a specified setting, not a universal constant for all populations. In his 1838 paper "Notice sur la loi que la population suit dans son accroissement", Correspondance mathématique et physique 10, pp. 113–121, Pierre-François Verhulst proposed $p^{\prime }=mp-n{p}^2$, equivalent to this equation with $r=m$ and $K=m/n$. He also noted that different assumed growth-retarding terms could describe the observations similarly while giving very different limiting populations. This is an early illustration of why a good-looking curve alone does not establish a unique long-term prediction.^[ref]

The logistic differential equation describes continuous time. The logistic map instead advances in discrete steps and can exhibit a period-doubling cascade and Devaney chaos; the monotone positive trajectories here cannot. Their connection through the forward Euler method is explained under Euler discretization of logistic growth, where a large step can introduce behaviour absent from the original differential equation.

Apply the chain rule to the squared loss, using the same pre-update weights throughout a forward and backward pass. At the output, $\partial E/\partial z_k=z_k-t_k$ and $\partial z_k/\partial x_k=g_k^{\prime }(x_k)$. ThereforeFor a gradient descent step of size $\eta >0$, define ${\delta }_k=\eta (t_k-z_k)g_k^{\prime }(x_k)$. This gives $\boxed{\Delta w_{jk}={\delta }_k{z}_j}$.

A hidden activation affects all output units to which it connects. ThusandThe learning rate is already contained in ${\delta }_k$ and hence in ${\delta }_j$; do not multiply by it a second time. For a sigmoid function, $g^{\prime }(x)=g(x)[1-g(x)]$. Thresholds can be treated as negative biases with an extra constant input and updated by the same derivative rule.

This is squared-error backpropagation. The forward pass computes activations, while the backward pass transports output-loss derivatives through the outgoing weights and the derivatives of the activation functions, yielding an error signal for every hidden unit. That reverse flow of derivatives explains the name backpropagation.

The sigmoid function is strictly increasing. Therefore the risky region is $\hat{\eta }(x)\ge \log \{p/(1-p)\}$. Its decision boundary is the hyperplanePoints on or above this score threshold are assigned to the risky class. For $p=1/2$, the right side is zero.

The feedforward neural network has three input coordinates, two hidden sigmoid function units and two output softmax function units. Every input is connected to each hidden unit, and each hidden unit to each output unit; each hidden and output unit also has a bias. A layer diagram isAlgebraically, with $x_i=({\mathrm{income}}_i,{\mathrm{age}}_i,{\mathrm{amount}}_i{)}^T$,with responses conditionally independent. The fourteen displayed weight and bias parameters comprise six input weights, two hidden biases, four output weights and two output biases. The one-hot encoding supplies targets $t_{i0}=1-Y_i$, $t_{i1}=Y_i$, and the categorical cross-entropy loss is the negative of $\ell ={\sum }_i{\sum }_c{t}_{ic}\log p_{ic}$. There is softmax non-identifiability under a common shift of both output logits, but the predicted probabilities are well defined.

The loop divides each feature by its sample maximum, putting features with nonnegative values on comparable scales and improving numerical conditioning for gradient optimization. If exactly one hidden layer must use the sigmoid function, use layer 3. Sigmoid derivatives can become small in saturated regions; placing it latest minimizes the number of subsequent gradient multiplications affected by this saturation, while the earlier ReLU layers retain efficient gradient propagation.