What is the significance of the recorded phone conversation between Bill Haydon and Toby Esterhase?

Actually, now that I review the scene and the scene that follows it, I think I get it.

The next scene is the one where Guillam goes into "the lion's den" to steal the log book for the night Ricki Tarr sent his telegram regarding a double agent from Instanbul. As he's leaving, Esterhase spots him and calls him downstairs to talk with "the adults" (Alleline, Haydon, Bland, and Esterhase).

So I take it Esterhase is trying to implicate Peter Guillam in Tarr's apparent defection. His "cabaret" is to plant the seed of doubt with Alleline. Alleline listening to the tape shows Esterhase doing just that. And the following scene, in which Alleline accuses Guillam of "consorting with an enemy agent behind my back," confirms it.

As for Haydon's reaction, I think in part he simply objects to Esterhase's pettiness. Compare the earlier scene where Control resigns: Haydon scolds Esterhase for mockingly waiving at Control and Smiley behind their back as they walk out. But I suppose it could also hint at Haydon's status as the mole and his efforts to manipulate Esterhase to serve his own ends.

Let's take these one at a time. (BTW: this is filled with spoilers.)

• What is the coincidence Esterhase suggests?

I think the word "coincidence" is unimportant. (I think that word choice is supposed to make it seem like he has more to say, but the truth is that the plot has already been served and so subsequent dialogue is unnecessary.)

The true significance of Esterhase's news is that British spy Rikki Tarr -- who is AWOL, presumed dead or turned -- has suddenly re-appeared. Very obviously, a spy org should be very interested to discover that such an agent has resurfaced.

• What is Esterhase's "cabaret"? and Why does Haydon take offence?

The truth is that Hayden is very interested in this news. Karla has presumably been after Tarr ever since Tarr learned from Irina that Karla has a mole in the Circus. Karla immediately captured Irina, but Tarr is still on the loose and knows enough to disrupt the Witchcraft plot. It is vital that Karla capture Tarr and learn what he has revealed to his superiors.

But Hayden does not reveal any of this to Esterhase, because Esterhase does not know Hayden is the mole and is not knowingly collaborating with him.

Hayden doesn't want the Circus to react to the news quickly (so that Karla can have the advantage when pursuing the newly-appeared Tarr). So Hayden immediately shuts down this line of conversation by chastising Esterhase, deliberately mischaracterizing as gossip this piece of significant news by referring to Esterhase's (news-gathering and reporting) activities as a "cabaret."

It is bluster and negative-reinforcement, designed to signal to Esterhase that the news about Tarr is unimportant and not worth repeating to anyone else.

• What is the significance of Alleline listening to the tape?

The point is to show that everyone is being actively and meticulously surveilled, and thus that the extreme measures taken by the actors to avoid detection are not only justified but actually necessary.

As you note in your own answer, this conversation happens shortly before Guillam's "lions' den" mission.

Guillam's mission is dangerous in an unusual way: unlike the typical spy who is operating undercover in enemy territory and so must conceal his identity, Guillam is operating at home. So the danger is not that he will be spotted or recognized by someone, but that someone will figure out what he's doing.

What is he doing? He's stealing confidential papers from the Circus' Archives. In service of that goal, Guillam arranges with a confederate to be interrupted by a phone call while he is in the Archives holding the confidential papers he wants to steal.

Have you ever arranged a "rescue call" before going on a date? You know: you tell a friend to ring your cell phone 15 or 20 minutes after your date begins, to give you an excuse to end the date early if it's going badly.

When your friend called, did they cook up some kind of believable story, or did they just say something like, "This is your rescue call." My guess is that they didn't bother fabricating a story, because they figured (1) your date would not hear the phone discussion / read the text message, and (2) you would make up something appropriate on the spot to sell the lie to your date.

Guillam's rescue call takes a lot more effort to set up than your friend invested in helping you ditch your bad date. Smiley's man places the call from an auto mechanic's shop, with the noises of auto repair audible during the call. Even though it's a "private" phone call, their conversation is 100% innocuous and provides a completely believable motivation for Guillam's subsequent observable actions at the Circus: asking for his bag (in violation of Circus security policy), rifling around in said bag for papers so he can make the swap, and then returning the bag.

And it's a good thing, too, because the Circus tests every element of this cover story in the minutes immediately afterward.

The point of showing Alleline listening to Hayden's phone conversation (as well as the "Mr Wu" song) is to make it clear to the audience that "standard rescue-call procedure" would have resulted in Guillam being caught and Smiley's operation failing. It makes the point that it's never safe to drop one's cover story: it must be solid, and Guillam must eat, sleep, and dream that cover story because his adversary is dedicated and diligent enough to see through anything less.

This is taken to its logical extreme a little later when we see Guillam abruptly evict his gay partner, on Smiley's advice to "tidy up" his own life since he will be under increased scrutiny.