Defamation is not fraud because no one has a legal agreement with the public by default to provide true facts only. If defamation was legal, there would be so many false accusations and fake evidence of the same, that most people would actually learn to think critically and not believe something until it has truly been verified. If a majority of people gain this skill, their critical thinking would be the predominant opinion, and anybody blindly believing things would have to be willfully blind.
