Why did Hermes leave immediately?

This is covered in the novel on which the film is based. In short, once you reach the Hermes the plan is to leave within the day. Mark states that this is an operational concern as waiting longer will delay the return (e.g. if you wait two days, you'll arrive back at Earth two days later and need two extra days of supplies).

I had no way to talk to Hermes. In time, I could locate the dish out on the surface, but it would take weeks for me to rig up any repairs, and that would be too late. In an abort, Hermes would leave orbit within twenty-four hours. The orbital dynamics made the trip safer and shorter the earlier you left, so why wait?