Sounds like a good plan. I did this a few years ago. Bought a few acres with cash. Cleared a home-site, put in a driveway and some other site work with cash. Then got a construction loan when I settled on a house plan and was ready to go. They used the land as the collateral, I didn't need "another" down payment. I ran my electric and drilled my well after getting the construction loan.
The bank was willing to loan more than just the construction (to cover a portion of the land), but I decided against this. Would be fine to do if you wanted to pull your cash back out.
Thank you Chris. That's pretty much exactly what I'm thinking of doing.