Switching lenders during escrow

[Jody Powell]

I asked this question in a different sub-forum and only had one reply, so I am asking here on the General Real Estate forum. I am looking for some perspective as to whether this is a bad move or not.

 I am a first time home buyer and have been working with a mortgage broker for the past 3 months or so. I am currently in escrow, and it dawned on me that I should have shopped around for my mortgage rather than just go to the broker suggested by my realtor. So I stopped in at my Credit Union and sure enough they offer me a better percentage rate (4.125 vs 4.375), no points and pay for the appraisal (saving me $545 in closing costs and a quarter of a percentage point).

I like my broker, but...

Purchase price of $205k. The loan is conventional, 20% down. ($164k loan)

What would you do? Any mortgage brokers or loan officers want to weigh in?

I feel bad switching but also feel that I need to be financially smart here.