Yes, every four or five years, a single government is elected that does every single one of the thousands of government functions. Voters don't perform a careful analysis of how efficiently these thousands of fuctions were performed compared to the previous ruling party. Since these governments also have the power to make various laws with sweeping societal effects, here's what actually happens:
Figure 1. "Think of how stupid the average person is, and realize half of them are stupider than that." – George Carlin, who didn't use the word 'median' to avoid confusing the average person.
Thus the incentive for the branch performing each function to do so efficiently is extremely small. A highly specialized and efficient agent is hindered by having to join, obey, and share in the outcomes of, this inefficient behemoth. In a libertarian system, the sole function and power of the government would be the enforcement of the NAP. This is best done at the local level. Any subjective laws passed will be based on local opinion, eg., laws protecting those incapable of consenting (children, the mentally disabled, etc.), laws against endangering others' safety, etc. The best performing government in each region can be elected, as opposed to the best one averaged across all regions. This is why libertarians want powers transferred from the federal to the local government.