Why is the audience sliding around in this scene from Le Temps retrouvé?

The source material is essentially a recollection of a life lived by an individual on his death bed. We are seeing these things through the lens of memory, so it is meant to be an almost first person account. There are other moments in the film where things move around for no apparent reason and the stage itself does not move in accordance with objective reality.

That particular scene is from "Swann In Love" by Proust and it was an attempt at capturing visually some measure of Swann's emotive memory of a musical experience from that story. I won't comment on whether it was successful.

The closest thing I can think at this moment that has this sort of movement might be in Wes Anderson movies where suddenly the naturalistic feel is broken in favor of a staged play: the Zissou movie and "Moonrise Kingdom" both take similar but more obvious breaks from naturalism.

They're definitely sliding.
Never seen anything like it.

At 1:37 the slim guy at the back & woman in the hat at the front can be used as 'markers' for the guy with fair curly hair & moustache. Also check the slim guy at the back against the candles They change positions in a way that cannot be explained by perspective zoom or a simple tracking shot.

Then at 2:08 it's clear as day; you can even see the inertia as they set off. You can see a couple of the SAs, camera left, affecting slight covering moves to try hide it as they set off; changing eye line, point of interest etc

Same for the final move of the pair of rows, you can see start & stop inertia. The two rows also move slightly out of sync, catching up & falling behind relative to one another, which becomes more noticeable as the camera tracking comes to a halt.

They were being pushed/dragged.

Best guess is they rigged up a track, same as you would for a camera, & mounted the seating on it - that's handy stuff to have around on set, it's not like they'd have to look hard to find something that can do smooth straight line moves on a movie set ;)
Camera track is very very smooth, but though a camera dolly weighs a heck of a lot [it's a 2-man lift just to drop it the 2" off the end of the track], it's nothing like 6 or 7 people to try to get moving without being able to spot it.

Note you never really see the floor, except for the barest glimpse, over towards the pink dressing screen [where none of the moves take place]

The camera is tracking too, to increase the sense of dislocation.

Oh... as to why... I've not the faintest idea - maybe the director was trying to inject some interest & movement into what would otherwise be a very tedious scene.
I'm not sure it really worked. Getting the camera moves smoother might have helped, even without the audience jiggling around in the background.

Ruiz is very creative in this film, and he gets his inspiration from Proust, who is equally creative at describing sights, scents and sounds.

There is a lot of 'sliding' in the movie, often involving the camera, sometimes set pieces or people. I think this refers to the effect of a kaleidoscope/magic lantern, which also makes an appearance in the movie (the scene is showing the sliding effect for a bit as well):

The 'magic' effects seem to happen whenever Proust wanders off in his mind, for instance when he's closing his eyes to listen to the music at the concert. Another good example is when he's reading a letter at a movie theater/restaurant:

His mind fills in what he 'sees', so as a viewer we get to experience more of his feelings. This also happens when he asks his maid about the roses he smells:

She denies, but he feels as though there's a huge rose in the room, and this is we get to see as well.

All in all, a very creative and thought-provoking film, you'd wish they'd make more of these...