Jokers pencil trick? He knew someone would get up to allow it to happen?

It's a response to the line of dialogue immediately before that, not an out-of-the-blue statement.

Give me one good reason why I shouldn't have my boy here pull your head off.
How about a magic trick?

That's not a reason, so the obvious response is to actually have the thug try to hurt The Joker.

It's been a number of years since I last watched the movie, but from my recollection The Joker is shown throughout to be good at predicting how people will behave. He clearly knew that this man, or one of the other people in charge in the room, would threaten him. Of course, if they don't, he goes an entirely different direction and the magic trick doesn't get mentioned at all.

Joker knew the gangsters would attack him, or at the very least threaten him, as he had just robbed them. He could work around that prediction.

Maybe the gangsters wouldn't approach him or be violent right away, but he was directly taunting them (saying they bought his expensive suit for example). It might take 10 or 15 seconds but eventually someone would tell a thug to get the Joker removed.

Or maybe they'd all be civilized. He could still flick the pencil over his shoulder and shout "ta-dah!". That's mostly what he did, he sowed chaos and improvised.

In his own words:

You know what I am? I'm a dog chasing cars. I wouldn't know what to do with one if I caught it! I just do things.

So basically you are wondering what would the Joker do if nobody got up and came to him.

The thing is, ever since the Joker stepped into the room he was playing his audience and at the same time pitching to them. For his proposal to be taken seriously, he needed the others to take him seriously first. Often a way to do that is to pick the meanest, most intimidating looking character in the room and own them. The Joker picked Gambol.

He begins by collectively pushing the mob's buttons by pointing out that they have lost a lot of money and possibly their reputation. He indicates that even though Lau has run away, he is still a potential snitch and a loose-end. The mob bosses already incensed are now seething. The Joker's looks and mannerisms don't help either. Somebody spits the word "freak" in disgust.

At this point the Joker is well aware that he is largely unwanted in this room. It is a safe-bet that Gambol (or one of the other bosses) would send one of their men to manhandle him. All he had to do was slip in the line "How bout a magic trick?" just before somebody walked up to him.

However impetuous his actions may seem, the Joker always has a plan. This is established time and again through the movie. One may say that it was his modus operandi. Here too, he had a script, a plan.