Refinancing a property purchased with a 203k loan

To my understanding, the 203k loan has the benefit of minimizing upfront cost to acquire the property,  but you get punished by the premiums over the long term. Is it possible to refinance after a few years to avoid that punishment, or are those costs already factored in the loan total? Thanks.