What operating system does Tony Stark use on his office desktop?

It's a fictional OS, designed by Philip Shtoll:

Not sure what the hacking dongle did exactly, but I guess it automatically accessed and downloaded certain files:

(you can watch Philip Shtoll's work reel for the movie here)

I cannot exactly recall the scene but I would bet it is one that they have pretty much made up for the film. You could probably knock one up similar based on linux if you had the skills and inclination.

The hacking dongle would, most likely, do nothing for real. If you had a secure system it would not allow anything on the dongle to run without the admin password and the admin password would be strongly typed and closely guarded.

However, consider the fact that actual hacking is quite boring to watch. It is mainly a numbers game combined with some specialist knowledge. Not very cinematic.

Hence film makers have to find ways of making hacking scenes quick and obvious to viewers, that is probably what the dongle is doing. It is a plot device that keeps the story going.

As mentioned in the other answers, the operating system itself is a fake, but in terms of "what could the hacking dongle do, coding wise"... well, anything.

The Universal Serial Bus used for these devices already has a proof of concept allowing a machine to be compromised just from inserting a compromised USB stick:

BadUSB – Turning devices evil. Once reprogrammed, benign devices can turn malicious in many ways, including:

• A device can emulate a keyboard and issue commands on behalf of the logged-in user, for example to exfiltrate files or install malware. Such malware, in turn, can infect the controller chips of other USB devices connected to the computer.
• The device can also spoof a network card and change the computer’s DNS setting to redirect traffic.
• A modified thumb drive or external hard disk can – when it detects that the computer is starting up – boot a small virus, which infects the computer’s operating system prior to boot.

It would simply be a case of knowing exactly what commands to send to get the data you need, and since it's Stark's computer accessing Stark's data storage via Stark's network, it's highly likely he already had those answers.

OSX or, if you want to use something very similar that is compatible on more types of computers .... I would say Debian 6 or Mint LMDE .... When Rhodey's suit reboots it looks VERY much like Debian linux.