How can humans travel to 12 different planets in Interstellar?

You are certainly right in that the 3 planets they visit are part of one system and the other 9 planets are part of other system(s), however this does not mean that they are located in entirely different parts of the universe. And in fact as you yourself state correctly, they could not control the exit location of the wormhole, it's a point to point connection with a single defined entrance and a single defined exit.

So the only logical conclusion is, that the other 9 planets, albeit in different systems, are not too far away from the wormhole either, but maybe just a "little bit" farther. Given that they took a 2-year journey to Saturn and they said it takes them months to reach Mann's planet (and their accessibility to comfortable hibernation technology), a possible journey of several years from the wormhole to any of the other planets/systems seems neither impossible nor out of the limits of their mission plan. So they could theoretically reach any of the other planets in "reasonable" time without employing the wormhole again.

However, they didn't want to, since on the one hand they didn't have the resources to check all 12 planets and on the other hand only those 3 planets in this single system presented promising results after the first investigation. Both of this is clarified in the dialogue when Cooper is briefed about the wormhole and the mission.

What might be a bit of a lucky stretch, though, is the fact that the Gargantua-system is the one nearest to the wormhole exit. But on the other hand we don't exactly know how far they traveled from the wormhole to Gargantua (given that the movie more than one time "hid" significantly longer traveling times) or where exactly that wormhole is relative to the other systems (it might very well be centered between all the systems).

They can't control their ship while they're in the wormhole, but once they're out of it, they can control it just as they could before entering.

As they noted in the film, a wormhole is basically a warping of space-time to create a sort of tunnel between one point and another. Once you're in, though, you can't control where you end up, with the only destination being wherever the wormhole spits you out at the other end. You'll end up at the same place every time, but you can't exit the wormhole without causing sever, if not complete damage to your vessel. The planets they were sent to explore were all located around Gargantua, and the wormhole spits them out at a specific point nearby. They can't use the wormhole to spit them out right near any of the planets, though, because the point it was open to near Gargantua is where it was pointed to, that's that. You get to the other end, and have to travel whatever distance you need to your designated planet.

They only go to three planets for the reasons noted in the comment on your question: they were the three that showed the most promise for being habitable for humans.

My assumption was that systems are defined by the central mass, a star, large planet, or black-hole.

Since Gargantua is a super-massive black-hole, it may have many other stars and planets captured into near and far orbits, and even extra-planar orbits (above or below the plane of Gargantua's planets). These far or off-plane systems would be much closer than typical interstellar distances (say, tens of astronomical units, like the distance to Pluto), but would require lots of fuel to visit individually.

So, the other planets and their systems could easily be other star systems that are very close to Gargantua, but much further or harder to reach than the 3 which lie close to it in a single plane of orbit.