In which District is the Hunger Games conducted?

Out in the wilderness, a different place each year.

The land mass that used to be called North America is nowhere near completely filled by the Capitol and the Districts; there are vast tracts of wilderness in between where nobody lives.

As @Mistu4u said, it might be unfair to set up the Hunger Games arena in one of the Districts - but then, who said the Hunger Games were fair? The Capitol, however, is just a city. There'd be no room for an arena there, let alone dozens of arenas used over the course of many years.

From the text of the first Hunger Games book (emphasis mine):

The actual Games don't start until ten because so many of the Capitol residents rise late. But Peeta and I must make an early start. There is no telling how far we will travel to the arena that has been prepared for this year's Games.

[...]

The ride lasts about half an hour before the windows black out, suggesting that we're nearing the arena. The hovercraft lands and Cinna and I go back to the ladder, only this time it leads down into a tube underground, into the catacombs that lie beneath the arena. We follow instructions to my destination, a chamber for my preparation. In the Capitol, they call it the Launch Room. In the districts, it's referred to as the Stockyard. The place animals go before slaughter.

Everything is brand-new, I will be the first and only tribute to use this Launch Room. The arenas are historic sites, preserved after the Games. Popular destinations for Capitol residents to visit, to vacation. Go for a month, rewatch the Games, tour the catacombs, visit the sites where the deaths took place. You can even take part in reenactments. They say the food is excellent.

From this we learn that the arena used for the 74th Hunger Games is half an hour's hovercraft ride from the Capitol - probably quite a long way, considering the speed of even the trains in Panem - and that arenas are scattered around at varying distances from the Capitol, and are popular destinations for Capitol citizens to stay for a month. You wouldn't go away for a month to stay in a place that was right on your back doorstep. The arenas couldn't really be described as 'in the Capitol'. In fact, they're presumably always far away from where any people live, to avoid the risk of having non-tribute bystanders involved in the Games (!).

I could not find any definite answer, but I guess it should be "The Capitol" where the hunger games were conducted. The reasons being:

1) If the dictator would have run the game in the Districts in front of the people, it might have caused another rebellion.

2) From wikia

The arenas were also a popular tourist destination for many Capitol citizens, who would spend months vacation visiting the arena from their favorite games, where they would be able to rewatch the Games, visit the sites of the deaths and even take part in reenactments.

So the arenas were a popular destinations to the capitols people. So it is assumebale they were created in the Capitol itself.

3) It would be unbiased to set up the arena in capitol because every district were participating in this game. Setting up in a particular district may attract rumors of being slanted.

Each of the districts was named 1,2,...,12 to make them separate from each other. Also the numbering helped the Capitol to administer them efficiently. Each district was unaffected from others in culture, living etc. and it was illegal for them to interact with each other. There are also more unique features to every one of them. Here is a short list of those:

District 1- Luxury (Great comfort or living)

District 2- Masonry/Stonework/Peacekeepers(not said officially in books, but later revealed.)

District 3- Electronics

District 4- Fishing

District 5- Power (not said officially in books, but later revealed.)

District 6- Transportation (not said officially in books, but later revealed.)

District 7- Lumber (Type of wood)

District 8- Textiles (Manufacture of cloth or woven fabric)

District 9- Grain

District 10- Livestock (Farm animals kept for use or profit)

District 11- Agriculture (The science or practice of farming)

District 12- Coal Mining

District 13: (Was thought to be terminated) Graphite Mining (Unofficially they manufactured nuclear weapons and among that housed and produced other weaponry).

More of the districts could be found here.

It's my belief that the arenas are created elsewhere in the land mass that was North America. The Capitol is one city, and the districts won't take up the rest of the country. I recall one of them being referred to the remains of a city, for example.

Also, to add something to Mistu4u's answer. District 2 was also responsible for providing a lot of the Peacekeepers.

District 13!

If it is so far away and the train travel so fast, the hovercraft will fly faster as they weigh less. They never specify that it wasn't and people would not think about District 13 as it was "destroyed". Also in the future like that and with 75 years of it, somebody would try to bomb it or attack the games. If in District 13 the people are underground and the upper part is clear they should be able to have the games there.