Amazing and wise answers, all. But, based on the hand-wringing I can sense in you (which is totally understandable), I have to go with the response that said you should just kiss the money goodbye. I only suggest that b/c you are clearly (and legitimately) concerned about the relationship with various members of your family. It's unfortunate that you have a family member who knows you have money and also knows that you value family more than you value your money. In some ways, I'm sure we all wish we had a family member we could hit not once but twice like that. But the chickens do come home to roost, so it's better to steer clear of conduct like that. The man is a low-life. It would be great to sue him. It's easy to tell you to do that from over here. But if you're more concerned about rocking the relationship-boat, you are going to have to 1) kiss the money goodbye, 2) forgive yourself for the time you burned by giving that much money away (b/c now you have to figure out how to recoup what you lost and that will take time), and 3) figure out a way to look the man in the eye every Thanksgiving. That can be done, but not easily.
Alternatively, and I think this is the real issue, you are going to have to get over your fear of confronting this issue head-on. You don't have to cause a big fight about it. But you do have to put your foot down, first with yourself (the bigger challenge) and then with him. Many families have a human ATM. For whatever reason, this man thinks that's who you are and he has no respect for you. I'm just telling you the truth. If the tables were turned, he wouldn't be doing all the soul-searching you're doing. There is a way to diplomatically point that out, expressing that you don't appreciate being exploited, and explaining that your investment goals and timelines are no less important than his. He won't grasp that, b/c he honestly thinks his idea of what to do with your money is superior to yours. He knows you as sweet, generous, and a push-over. There's nothing wrong with being meek. But it's absolutely possible to be confident at the same time, and being disrespected by another man in your family is more than a little obnoxious, in my opinion. And I'm not even a man. If you are going to do something about it, you should schedule a polite and respectful meeting with him privately at a time and place where none of the wives and sisters are around or even aware. Sit him down, tell him how he is messing up your life, and graciously put him on a timeline, explaining that you have already spoken to an attorney about the situation. If he has a fit (which he may), maintain your composure and explain that HE has left you no other choice.
BUT if you can't see yourself doing that and if all the women in the family are going to create merry you-know-what if you do it, then take a deep breath, apologize to your wife, kids, and grandkids, and let it go.