Selling a flip with a bade house next door

We currently have a flipped property we are trying to sell. It is in a great neighborhood and many homes are starting to be flipped and updated. We have only been on the market a few days but all our feedback is positive except for the part where they say buyers did not like the next door neighbors house. We already have it priced below what most homes are comped at. Am I freaking out over nothing since we have had so many showings in two days or is there anything I can do?