Are you advertising what you want for the Option fee or do you tell them when they call? I don't know what a regular SFH sells for in that market (I just know the COMPS you said about other townhomes) but if you can buy lots of homes for under $100k I would think that people that can't qualify for a traditional mortgage may not have the 3.5% for the down.
In my market, a home like that would sell for $200k on a Lease Option and the down would be closer to 2%.
If I were you, I would consider a smaller down to get more available buyers. That doesn't mean your getting lower quality people, you'll still be able to pre-qualify them and pick the best one.
Most people don't walk in and negotiate a down payment in that situation, especially because they're choices are limited. Buyers with bad credit may be a little timid to negotiate, even if they only need to ask for a $500 difference. I would think that $2500 down would be more realistic, based on the sale price. Just my opinion.
Jeff