Why did the Reavers attack other planets instead of each other?

Personal theory-

They are like a different species from humans. As other carnivorous animals, they don't eat each other. As some carnivores live in a pack they also live in a pack. They didn't even take interest in dead humans similar to some scavengers. They are anti-social and brutal but still they show some intelligence as driving ships,so they are not dumb and they have little intelligence to know who are their one kind.

They are similar to other Movie characters like were-wolves, vampires, Aliens, predators etc. who also exhibit similar characteristics.

Sadly, there's no explicit explanation in canon.

What Reavers do

The Reavers are shown to cooperate from the first time they are observed in-universe:

0.1% of the population of Miranda [...] acted conversely to the Pax gas agent, causing them to go insane.

The other 99.9% die by complete inactivity. All non-automated responses from Miranda cease. Alliance scientists come to investigate. The last member of this scientific exploration records the message that Serenity's crew finds on Miranda.
The message ends with a pack of Reavers busting through the door and killing the last scientist. This was a coordinated effort. The Reavers were out for this kill. They knew someone was in there. They expended effort to break in. They risked their lives in a fight.

As for

mental status to be able to pilot starships

The Reavers' scenes closest to a cooperative effort in working on ships are:

• A booby trap on the convoy with the "survivor". This trap was set up off-screen. However this establishes that Reavers don't only use but also manipulate technology.
• When Serenity sneaks to Miranda: Two ships pull a third apart. One could argue that Reavers scavenge ships for parts.

How to become a Reaver

It is also shown that normal people may become Reavers far from Miranda and possibly without Pax: The crew of Serenity loots a powered down convoy and finds a lone survivor who repeatedly stutters

No...
Mercy

Brought up for inspection by an Alliance cruiser, the crew is only allowed to go on, because Mal helps hunting down the "survivor", who turned into a Reaver.

Mal has something like this to say about him:

[The Reavers killed everyone onboard and] made him watch. [...] He took the only course left to him: Becoming one of them.

If Mal is right, being a Reaver is a mental state of aggression with no other option than following the group pressure of violence against weak and fleeing targets.

What Reavers don't do

Mal seems to understand the Reavers, as he repeatedly makes correct predictions about their behavior, e.g. Mal's call not to try and run when they crossed path with a slowly floating Reaver ship as to not provoke them to a hunt.

Reaver cooperation seems limited. Their ships are deranged and regularly with defunct core containment. They don't exactly care for each other or their own well-being.

Reavers don't stop. They take no cover and they don't hesitate. You can see this from the bank heist in the movie. At its end River remarks

He wouldn't stop.

This is kind of ironic, because in the end they find out that Reavers are the inhabitants of Miranda that didn't stop.

My hypothesis

Reavers extend their influence and attack other planets, because they are driven by aggression. However Reavers aren't insane as in "not able to think straight". I'd characterize them as utterly violent psychopaths. They also don't seem to be egoistic but always work towards the common goal of violence.

Maybe they are trying to get back at the Alliance without knowing whom to target specifically. Alternatively they may be trying to turn normal people into Reavers.
Notice River's pleas to the dead people on Miranda to move. Maybe aggression (which is what Pax was supposed to suppress) was the only way to keep people going and they couldn't stop afterwards.

I'm going to go out on a limb and say that Whedon wasn't able to fully explain the back story to the Reavers. Whedon originally planed on stretching the universe of Firefly over years, including a video game. If you watch Done the Impossible (a fan documentary), Whedon says the plot of the film was intended to continue from where the show left off but also be easy for non-fans to understand. In full, I don't believe there is a way to answer this question.

While Wheedon provides no real answers here, nature may bring some insight into view. I would suggest the Reavers were not "insane" so much as hyper-sensitive and overly aggressive. They would probably still have a hierarchy and order to their way of life. There would still be some type of "chain-of-command" where as to keep them in order. In the movie Riddick, they "keep what they kill" - seems this might apply here as well?

It's almost like pheromones. If you don't exude a certain secretion or smell you are perceived as in this case a target. That's why they don't (assumption) attack each other. The opposite effect for zombies. The living put out a certain something that attracts the walking dead or undead as the case maybe.