How did Kirk sell his house?

He didn't, humans have changed faster than our language, as this conversation between Quark and Jake Sisko in DS9 You Are Cordially Invited... shows:

Jake: I sold my first book today.

Quark: Really? How much did you get for it?

Jake: It's just a figure of speech. The Federation News Service is going to publish a book of my stories about life on the station under Dominion rule.

Quark: And they're not paying you?

Jake: No.

Quark: Well then, you have my sympathies. And the first round of drinks is on me.

Jake: Really?

Quark: No, it's a figure of speech

To my knowledge, the exact mechanisms of land ownership are never properly explained in anything regarded as canon, but you can be fairly sure it didn't involve money.

Possible answers consistent with both quotes (not necessarily mutually exclusive):

1. When Kirk states that money doesn't exist anymore, he means physical cash. Most of the Federation economy is subsidized, so no one HAS to work to survive (note: freeloaders would get 'therapy'). The rest is handled by a residual digital-money system (note: avarice also invites 'therapy'), and obviously Kirk cannot spend his 'bits' in the 20th century.

2. People in Starfleet don't need to use money, Starfleet takes care of everything within itself (just like a military service). The Federation has an economy with money (heavily regulated), but Kirk hasn't needed access to that in a while.

3. 'Sold' is just a carry-over word. There was an ownership transfer, possibly including an exchange with something else of value.

4. There IS a money-based economy involving non-Federation commercial interests (Ferengi!), and Kirk sold his house (which is not necessarily on Earth) to someone outside the Federation.

How money and economies work in the Star Trek universe has never been properly explained in any canon material. StarTrek.com actually ran an article a few years ago in which they addressed this hole in the universe, and admitted that there are far too many contradictory statements in the episodes & movies to come to any conclusion. So while there's some truth and some possibility to the explanations given in the other answers, the unequivocal answer is that there isn't one.