Mortgage Question: Second Home vs. Investment Property

[Delin W.]

Considering purchase of a PUD unit (townhome, not condo-tel) in a resort area and would like some advice about what the exact rules are governing second home and investment property loans.

We intend to use the property ourselves as much as possible (8 long weekends per year, plus a few weeks when we can), but plan to rent it via VRBO when we are not there. I will manage the rentals and not use a management company.

We will claim the rental income to the IRS, but do not expect it to be much; we may or may not break even. 

Can anyone point me to the exact language Fannie and Freddie use when defining owner types and loan types? None of the loan officers can point to the exact language. They basically tell me I can do either loan, but getting an investment loan would be the safe route. I agree, but darnit, I would not buy this property if I did not intend to use it myself as a second home! Any advice out there?

TIA