Why can't I get a gov loan for a property that has 2 houses?

[Addison Ricci]

So I have my first investment property under contract and I was supposed to close this Friday.  My lender informed me yesterday that they couldn't close on this property because there are 2 separate houses on 1 plot of land.  They each have their own meters (which I need for rentals) but they are deeded together on 1 parcel.  The lender said Fannie Mae/Freddie Mac won't deal with these kinds of properties so they can't close.  Does anyone know if this is actually a thing?  If the houses were connected and it was a duplex I don't think this would be a problem.

I'm working on finding a portfolio lender, but that means that i'll have to put more money down and probably have a higher interest rate.  It's not the worst option in the world, but I wanted to make sure the lender actually knows what they're talking about.

Thanks!