Maybe your numbers are wrong.

Wrong!

WrongDo you have a blog that earns you money?

The crux of the issue is how do you get a sub-contractor to show up to the next job after you penalize him/her for his/her wrong doings on that job (full rights under the contract)?

Our opinions on these forums are just that, and many of us are wrong.

Even if I do everything by the book any hint of wrongdoing can cause a lawsuit 

And while it is true that the seller's agent owes you some level of duty (e.g. taking some reasonable steps to ensure that the seller didn't disclose false information on the seller's disclosure), I don't see any obvious wrongdoing from the short facts you gave.If you want to pursue this, I again recommend that you sit down and talk to a lawyer.

But sheesh that's just so sad and it does nothing but cause more pain and suffering for everyone else who isn't even involved and has to pay for the wrong doing of others, ie increased insurance etc.

Someone thinks they may have a case but doesn’t provide any useful info beyond the suspicion of wrongdoing. 

This is standard business practice - just look up how many settlements the SEC makes annually where companies pay millions in penalties WITHOUT ADMITTING to any wrong doing.- Cash-for-Keys is another example of this!