Why had 5 1/2 hours passed on the ship?

You're assuming a never-stated correlation between time differences. The examples are not a fixed, linear 1 minute = 1 week pattern. One second could be a year. Another second could be a decade, another just a minute, etc.

As you said:

After the clockwork androids are stopped the Doctor is trapped in Versailles because all the time windows are broken. Then Reinette brings him to the fireplace and he returns to the ship. Rose tells him that 5 1/2 hours had passed on the ship. From the time the Doctor broke through the glass in the ball room and he returned to the ship via the fireplace, only few minutes had passed in Versailles timeline. But time moves slower in the ship timeline so only few seconds should have passed instead of 5 1/2 hours.

You also said:

Third time- couple of hours on the ship, years passed in France... Reinette is 23.

First off, what made you think hours had passed? It seemed just a couple minutes. This is another example of lack of "typical" continuity. The last example towards the episode's end was less than a minute for the Doctor on the ship and twenty years in Versailles.

And as your first quoted observation shows, the episode has already proven that your assumption is not the case. Your examples of the discrepancies are proof of this.

How often does the Doctor and / or TARDIS show up in a time other than anticipated? A LOT. One can safely assume this is because time is not easily manipulated or plotted. Also, the ship the Doctor and his companions landed on was "generating enough power to punch a hole in the Universe" producing a "spatial temporal hyperlink" (which he made up instead of saying "Magic Door"). An anomaly like this wouldn't have to be constrained to our current understanding of time and space.

As to why he said "Always wait 5 1/2 hours?" It seemed like a joke to me. He was just excited.

As a postscript, this is also an episode the Doctor advises "You never want to listen to reason."

There is a loose wire here :: sonic the "mantle" then smacks it:: most likely the time dilation varies based on the energy consumed in the transfer of matter between timelines