What video game was Groot playing?

No, it wasn't Galaga. The game he was playing was Defender.

From Movie Web article,

The thing is, "Arcade Defender" already is a video game from the 1980s. How the game made its way into space has yet to be explained, but it's still a fun reference for those who have played the game in the past. The "Arcade Defender" logo was already similar to the Defenders logo, so it really didn't take much to make the "Arcade Defender" logo look like the comic logo. All they needed to do was make the edges a little sharper and the Easter Egg would be good to go.

From Screencrush,

Throughout the entire film, Teen Groot is obsessed with a handheld video game. In a brief cutaway, we see that he’s playing Defender, an Atari classic where your job is to protect Earth from alien invaders. Obviously the game is an allusion to the plot of Infinity War, but the name defender also evokes the Defenders comic which, unlike the Netflix series, traditionally features Infinity War characters like Doctor Strange and the Hulk.

Does this game have any relation to a previously shown video-game reference?

The game Galaga also appears to hint at the plot of The Avengers. The same thing Defender depicts for Avengers: Infinity War.

Or, perhaps it symbolizes something specific to this Avengers film?

This game seems to be relevant to the plot of Avengers: Infinity War. The task in this game is to protect Earth from alien invaders (as explained in above article) and they do go to Earth to save it from Thanos and his army.