What was Mann's motivation?

First of all, we have to remember that Mann was a coward. Upon arriving on his planet, he realized it was uninhabitable but didn't want to die alone, and as such sent out a beacon to say he had found a Goldilocks habitat, so a rescue mission would be launched to find him.

Cooper, believing that they had found a suitable habitat for colonization (based on Mann's lies) was determined to take the shuttle through Gargantua and back to Earth. For all intents and purposes, at this point they believed the mission goals to have succeeded.

Cooper was going to leave them on the planet and return home.

However, Mann was lying about the planets habitability: it was, in-fact, uninhabitable. Mann was acting in self-preservation, and his options were either to somehow gain Coopers trust, assume the pair of them could overrule or overpower the rest of the crew, and return to Earth having failed the mission: or, to kill Cooper when he is vulvernable and unsuspecting. His plan from here is largely undisclosed, and he was possibly only acting spur of the moment, killing Cooper when the opportunity arose to stop him abandoning the remaining crew on an inhospitable planet.

Because Mann lies, it is unclear whether he intends to return to Earth alone in the shuttle, or (as he broadcasts to the crew) to "complete the mission" and find the last hospitable planet to start a colony. His cowardice would indicate the former, however.

Basically, cooper was going to leave the crew abandoned, and Mann saw the opportunity to eliminate him. I don't think this was premeditated, but when Coop asked Mann to escort Coop to the monitor stations, the opportunity was too good to pass up.

Dr Mann's motivation is survival. He knew no one would save him if he was on a useless planet, so he activated the beacon to get help to come.

His reasons for trying to kill his rescuers, either by straight up murder or abandoning them on the dead planet is to protect his pride. He doesn't want people to know he wasted time and potentially millions of lives on earth by getting the only hope of humanity's survival distracted.

One of the key plot points is the juxtaposition of Mann and Cooper. Cooper being reluctant to go but willing to die for the betterment of humanity and Mann being eager to go but unwilling to die even at significant cost to humanity.

Because Cooper wanted to go back to earth once he realised that they weren't meant to save earth, but save mankind from extinction.

Mann realised this and wanted to stop him as he thought there was no hope for mankind on earth. I think he even mentioned this in the fight scene once. I need to see it again to confirm though.

if mann would have been sane, and just joined the crew back to earth, then TARS would not have been able to collect the quantum gravity data from the blackhole.. and cooper would never have communicated with murph, and plan A would never have succeeded and humanity would have died. Also since the first half of the movie happened, and Cooper had communicated through time using gravity, he was part of a closed time loop.. hence Mann had to behave the way he did :)