Good Morning, Bigger Pockets world!

So I currently am on orders to Fort Polk, I wanted to build a duplex the problem comes that no builders will utilize the VA loan and stated that I needed to lend through a bank and then transfer it over through the VA loan. I understand the reasoning, that builders are building out of their own money in some occasions but I never heard of this happening. I already have been pre approved through the VA loan and have a pre approval letter. I was planning to put down around 10,000 or so as well. Is this common when building a duplex? What other process can I use? Is this ethical? Can i just sign a contract or use the VA loan to purchase? Kind of hesitant, cause I know the VA loan guarantee certain things before a person can move in. I don't know if its weird business being conducted, but certain inspections have to be approved. If I go through the bank that would be eliminated.

Any feedback or advice anybody can give me on this situation.