2/5 year rule for primary converted to rental

As I understand it, if you live in the house for 2 years out of the last five before you sold, you get to avoid capital gains tax, so long as you bought as a primary residence, but the IRS doesn't care if it's the first two years or the last two years, so long as it was purchased as primary residence and not an investment property initially.

So what happens if you lived in the property the first 2 years, then rented it for 4, and move back into it for 2. Would you pay capital gains tax on the pro-rated portion or be exempt from capital gains (other than depreciation recapture)? Source please...