I have plenty of equity built up in my primary residence and was planning on using that as a down payment. 

I'm obviously trying to refinance the $170k that I owe (around 73% LTV on $230k), and then use the private loan to go buy another in cash.I spoke to a lender who told me that because the house is a duplex and not a primary residence, the best rate I could get on a refi will be around 5.75% on a 30-year fixed.

I will have property management and plan to be the primary contact person and my partner will be second.

The reason I suggested for you to look into 1031 is if your profit from selling your primary residence is more than $250k, then you'd still owe taxes on the difference. 

I set a few goals for myself this year after buying my first primary residence, and up until now I’ve been scrambling.

I've bought and sold a few primary residences, so I'm familiar with the process of buying/selling a home, and I've read some books and lurked around the forums a bit.

VA Loans must be used for primary residences and there is zero leeway. 

Two years is a long time for a hard money loan, so I'm also looking at some other options.One of the primary reasons for going 40+ units is for the budget to pay a property management company.

I have my primary mortgage in a HELOC (HELOC in 1st position with variable interest), interest only.

@Timothy Albright yes for fha you would have to make it your primary residence but it doesnt have to be your 1st loan. if you have equity in your current home + $15k liquid you can definitely do a flip/hard money rehab.