 Solution

With identity hidden activations and no biases, the pre-softmax map is a product of weight matrices. For any nonzero scalar $c$, multiplying the incoming weights of one hidden neuron by $c$ and dividing its outgoing weights by $c$ leaves that product, every output probability, and the cross-entropy unchanged. Every minimizer therefore belongs to an infinite continuum of equivalent parameterizations; more generally, invertible changes of hidden coordinates and their inverse in the adjacent layer give the same network function.

I would use batch_size=1. The resulting stochastic-gradient noise helps move along flat nonidentifiable directions and escape saddle regions, whereas full-batch gradient descent is deterministic and can stagnate in this highly nonconvex, singular parameterization.