Why is Oogie Boogie associated with gambling?

Why does Oogie Boogie have such a major connection to gambling?

This is a matter of character exposition. There's no need for Oogie to be a gambler, but it add flair/style to his character. Making him a cut-and-dry villain wouldn't work in a setting where everyone in Halloween town looks like a villain. He needs to stand out of the crowd in a way that the audience sees him as a narrative villain.

Most character exposition we get from Oogie Boogie is through his song:

Oh, the sound of rollin' dice
To me is music in the air
'Cause I'm a gamblin' Boogie Man
Although I don't play fair

It's much more fun, I must confess
When lives are on the line
Not mine, of course, but yours, old boy
Now that'd be just fine

This encapsulates why Oogie's gambling defines his narrative villainy: he seeks to harm others. He claims that this is a matter of a dice roll, but we see him change the outcome of a dice roll when it is not harmful to Santa.

Compare this to the other Halloween town citizens. While they do things that appear horrible to us, they always have good intentions. Every toy they built was truly intended to be a gift.

Is this a reference to anything in particular?

The Gambler is the defining trope here, but interestingly, Oogie subverts it. As the trope page points out, Oogie's subversion is particularly reminiscent of Joker from Batman.

The trope page specifically lists Oogie:

Writer and producer Tim Burton and director Henry Selick's The Nightmare Before Christmas has Oogie Boogie, who is shown to base his actions off the roll of a die when torturing his captives. He also has a couple of game-based traps in his dungeon: a giant spinning roulette platform, monolithic cards with swinging swords, one-armed bandits with guns, and the base of spinning propeller blades resembles an 8-ball. Like The Joker, another rare example that manages to be outright evil - he even admits to being ready to cheat in his Villain Song.

Gambler trope characters are inherently defined by them sticking to dice rolls as opposed to predetermined morality. However, do note that Oogie Boogie subverts this by changing a dice roll when it rolls in Santa's favor (thus making him a clear villain).

Much like the Joker, Oogie is a character that pretends to be chaotic neutral ("I leave it up to chance") but in reality is chaotic evil ("I inflict cruelty").

However, I doubt this is intended as direct reference to Joker. It seems more in line with adding a touch of flamboyancy to Oogie's character in order to make him stand out from than the "classic villain" style characters that inhabit Halloween town.

I thought it was just a reference to the phrase "The Gambling Bug" -- this is the usage I'm familiar with:

The point is, I’ve never felt the sting of the gambling bug, probably because nobody under the age of sixty has said the phrase “gambling bug” since Sinatra opened at the Desert Inn. (emphasis mine, source: https://www.myentertainmentworld.ca/2015/12/how-i-win-at-casinos/ )

and another example:

Gambling has been in the news a lot recently what with the amount of people 'addicted' to it or teetering on the edge of addiction. I'm sure most of us know someone who just can't help themselves whether it be horses, greyhounds, football accumulators or even online bingo. As someone who doesn't have the gambling bug and who has never put a bet on his life, I have a lot of sympathy for people whose lives have been ruined because of their gambling habits https://not606.com/threads/gambling.370303/page-2

-- and it was apparently literalized as a Looney Tunes character: https://en.wikipedia.org/wiki/Early_to_Bet

The narrator introduces the Gambling Bug, demanding he stand up so everyone can get a look at him. Three examples are then offered, showing what happens should this Gambling Bug "bite" someone, giving them "gambling fever".