Currently my family and I live in a starter house. We already have a lot and plan on building a house when lumber prices hopefully fall a bit more, so we can wait for a year or two. 

Our current primary residence is worth $100k in current market conditions and we have $50k left on the house.  

I came across a nice duplex in a college town and am thinking of pulling money out of my primary residence by refinancing to buy this rental duplex. 

When we build our house, we will most likely hold on to our current house as well and add it to the rental portfolio.

Would you recommend refinancing to have cash available to purchase the duplex with it?