At this point, your documentation will be key. You will need to show the banks that your duplex is cash-flowing. You definitely want to save up all of the money you can so that you can show solid reserves for the duplex and primary residence.
Would it make sense for you to keep the primary residence as a rental when you move out?
@John Pelaez - I had a similar situation happen to me when I purchased my second property. I had to have a tenant sign a lease to my place to prove to my lender that the rent would in fact cover the mortgage on the first place. This is something that you may have to do.