Financing for an fha property- third house

I have a single family with no mortgage but has a heloc. I also have a duplex with a conventional mortgage that I live in. I was talking to a lender about buying a four unit and he said I can’t use fha for it. He said I could only go down after buying fha, as in if I had bought a four unit then I could get a triplex or duplex. Has anyone else heard of this rule? Also the houses I’m looking at are less than 100 miles away. Could I be eligible for a duplex since I’m not going up in units but just staying the same? This whole process has been rocky and I want to get out of the house I am in now. Please help.