What tune does Tyrion Lannister like to whistle?

Do you mean The Rains of Castamere?

The Rains of Castamere is a famous song in Westeros... [It] immortalized the destruction of House Reyne by Tywin Lannister.

The Rains of Castemere went on to become very popular with soldiers of the Westerlands, becoming an "anthem" of sorts for House Lannister. This extends to the point that even Western soldiers sometimes refer to it simply as, "the Lannister song".

Tyrion whistles the tune to the song when he arrives at the Small Council in King's Landing for the first time. He does it again while on his way to visit Shae, when he is surprised to find that Varys is there with her.

Here's an instrumental version. It's also sung in Season 3 and 4, there's an episode named after it, and it plays during the Purple Wedding. One instance of the whistle can be found here (albeit a bit out of tune):

The other answers are mostly correct but for a bit of extra trivia the first tune Tyrion whistles in season one (when he and Bronn are leaving the Vale and before they are approached by the Hill Tribes) is actually the main theme from the first movement of Beethoven's 3rd symphony 'Eroica'. I doubt this has any higher meaning in the show and is simply a random tune the actor knows but the music was written by Beethoven with a dedication to Napoleon Bonaparte, though he removed this title due to the composers outrage upon hearing that the french general proclaimed himself as Emperor and instead gave it a tile meaning 'heroic'. So perhaps Tyrion is deliberately whistling a heroic tune, though the music itsself is separated from Westeros by a few hundred years of cultural advances and refers to a completely different universe.

I don't watch the show, but since you asked for book-based answers, here are a couple of songs that stick in Tyrion's head consistently in the books. (The Rains of Castamere, as mentioned in the accepted answer, is an iconic Lannister song, but not at all specific to Tyrion, and given his ambivalent attitude towards his House, he probably doesn't spend much time whistling it, especially after the events of Season 4.)

"The Seasons of My Love"

This is the first song we see Tyrion having a special liking for whistling, as he associates it with his lost love Tysha, his first wife. From the first book (emphasis mine):

He began to whistle a tune. [...] He resumed his whistling. "Do you know this tune?" he asked.

"You hear it here and there, in inns and whorehouses."

"Myrish. 'The Seasons of My Love.' Sweet and sad, if you understand the words. The first girl I ever bedded used to sing it, and I've never been able to put it out of my head." Tyrion gazed up at the sky.

_{A Game of Thrones, chapter 42 (Tyrion)}

We know some of the words to this song, but not all of it. The lines we know are (the separate stanzas here not necessarily adjacent or in the correct order):

I loved a maid as fair as summer
with sunlight in her hair.

I loved a maid as red as autumn
with sunset in her hair.

I loved a maid as white as winter
with moonglow in her hair.

Symon Silver-Tongue's song

There's also the song penned by the minstrel Symon Silver-Tongue in his attempt to blackmail Tyrion. Although Symon himself comes to a swift and deadly end at the hands of Bronn after this meeting, the song sticks in Tyrion's head and he often finds himself whistling, humming, or singing it later on, including during his trial and while killing Shae. The one verse we (and he) know goes like this:

He rode through the streets of the city,
down from his hill on high,
O'er the wynds and the steps and the cobbles,
he rode to a woman's sigh.
For she was his secret treasure,
she was his shame and his bliss.
And a chain and a keep are nothing,
compared to a woman's kiss.

with the refrain:

For hands of gold are always cold, but a woman's hands are warm.

"The Rains of Castamere" is his father's song and the tune he hums is from the first girl he married, after they got married and slept together she hummed it. Then Tywin found out and made his people sleep with her while Tyrion watched and went last.