Why do the wildings attack the wall where they do?

Mance Rayder in intending to move the entire Wilding population south to avoid the oncoming winter, which is presumed by many to be likely to be long, and accompanied by the reappearance of 'the others'. He has an army estimated as 100,000 strong.

The remaining other forts along the wall are deserted yes, but its also presumed that the tunnels through the wall at these points has been filled with stone & ice. So yes, Mance could get people to climb over the wall here (as indeed we saw last season), but he cannot move vast quantities of people over the wall, which would ultimately have to include children and the elderly of his population. Even getting 100,000 over the wall seems unlikely - we must presume he only sends his most capable raiders on that route.

For Mance's invasion of the north to work, he either needs to tunnel through the wall or capture one of the occupied forts. He knows that the force of Crows is historically light in numbers, even if he doesn't know their true weakness - and that his army will also have to face the army of the Warden of the North at least, if not the Seven Kingdoms. Attacking and getting his army past Castle Black is the key first step in his invasion.

This is better explained in the books.

Mance original plan was to attack the Wall at the same time from two sides. He would first attack from the North. He would expect to have all the Brothers of the Night's Watch fighting him at the North side of the Wall.

While he and his army were attacking from the North, his team of scouts would attack from the South (in Castle Black) in order to break their defenses and raise the gate that would allow them to pass through to the South.

Mance didn't know that Jon had escaped and ruined their plans.