Physical problems with the building or a bunch of undesirable tenants in the building.

It would be worth studying up on that technique.Because you know the area so well, presumably you have a solid understanding of which specific neighborhoods are most desirable (and un-desirable)--use that to zero in on properties in the most desirable areas.

When the lake started to recede and the salt water level intensified until it killed off all the sea life and slowly ate away at paint, metal, wood or anything else in its way, the Great Salten Sea became an abandoned ghost town and refuge for motorcycle gangs, drug runners, illegal's and just about any/all undesirables that could survive there. 

For example, I have bought a property in an undesirable area, and noticed that the rent and value hasn't changed much.

And it we are, on none rent control unit, can we just raise the rent really high so the undesired tenants would go away?

I have given a few agents calls, but it seems difficult to weed out those undesirable characteristics over a phone conversation.

We have our fair share of undesirables, but you can usually weed them out with proper screening!

Seriously, grime and dust and dirt can make a place feel pretty undesirable - you don't even always notice it when it's there, but you will notice it when it's gone because the place will just look, feel, and smell nicer.- On that same note, one of the other most important things is also to make the place feel light and bright.

You mentioned the undesirability of refinancing because of investment rates and LTV.  

For 2 bedrooms in my market, there are a lot available, however, most are in the undesirable part of town.