TODO confirm, but looks like it, e.g. E. Coli starts DNA replication before the previous one finished.
Because DNA replication is a key limiting factor of bacterial replication time, such organisms are therefore strongly incentivized to have very minimal DNAs.
Power, Sex, Suicide by Nick Lane (2006) 7 "Why bacteria are simple" page 169 puts this nicely:
Bacteria replicate at colossal speed. [...] In two days, the mass of exponentially doubling E. coli would be 2664 times larger than the mass of the Earth.Luckily this does not happen, and the reason is that bacteria are normally half starved. They swiftly consume all available food, whereupon their growth is limited once again by the lack of nutrients. Most bacteria spend most of their lives in stasis, waiting for a meal. Nonetheless, the speed at which bacteria do mobilize themselves to replicate upon feeding illustrates the overwhelming strength of the selection pressures at work.