That character was Rankin and Dexter did kill him out of grief rage. It was not like his previous psychopathic planned killing but a sociopathic rage kill.

Even his Harry's hallucination tells him that that's the first human thing he'd done since his wife's death.

And he is not the only kill outside Harry's code, there are many, from dexter.wikia.com:

Kills Outside The Code

Every season (after Season Two) saw Dexter kill at least one person that didn't meet Code, intentional or otherwise.

• Season Three

  Oscar Prado - Killed during a struggle at the home of a drug dealer. ("Our Father")

  Nathan Marten - Pedophile, killed for taking photos of Astor. ("The Lion Sleeps Tonight")

  Camilla Figg - Mercy kill, at her request. ("Easy as Pie")

• Season Four

  Jonathan Farrow - Killed in a 'kill room' and dumped him in the Gulf Stream. Dexter believed him to be a killer but it was actually his assistant. When Dexter found out that he was innocent, he deliberately broke Farrow's blood slide. ("Slack Tide")

• Season Five

  Rankin - Killed on the spot, out of anger. ("My Bad")
  Stan Liddy - Killed in self defense, although Liddy may have killed somebody at some point since he was a former police officer. ("Hop a Freighter")

• Season Six

  Norm - Killed when he demanded money from Dexter, although Dexter knew that the gun being held on him was not loaded. ("Nebraska")
  Alberto - Killed for robbing Cuban immigrants and threatening them with a gun. ("This is the Way the World Ends")

• Season Seven

  Clint McKay - The first and only innocent victim to receive a ritualistic kill (wrapped in plastic and dumped into the ocean). ("The Dark... Whatever")

• Season Eight

  Andrew Briggs - Killed during a struggle (which he caused). ("A Beautiful Day")

It might be debatable if it was real since the man is not given a name and never shows up again. And I don't remember if he hallucinates Rita, but the episode is filled with flashbacks to when they first met and started dating.

However, as the show progresses, it becomes clear that Dexter is not really a 'text book' (by tv standards) sociopath, but a traumatised introvert that actually feels VERY deeply about others. His reaction to Rita's death, along with what he does for another character featured in season five, is the solid proof of Dexter's awakened humanity, but that in itself is what in part leads to Dexter having more problems in later seasons, where he doesn't always stick to the code or makes mistakes. So in a way this is like a lead-in for that idea. Even Rita's death was a byproduct of Dexter torn between the idea of wanting a companion to understand him/be himself with (something that was instilled in him the moment he learned of his brother, which even in that moment, Dexter choices Deb over Brian) and wanting to save the lives of children, since Arthur Mitchell totally defies Dexter's code.

Note: there are couple episodes though that are more surreal than others, such as the episode Left Turn Ahead in season seven...

Dexter Wikipedia - Season 5.01 My Bad - Plot Synopsis:

The boat runs out of gas and Dexter pulls in. In the store, he watches as an angry man repeatedly rings the service bell to get the key to the bathroom. When nobody comes, the man searches the desk until he finds the key himself, and turns around, taking offense to the way Dexter is staring at him and angrily calling him a retard for doing so. Offended, Dexter follows him into the bathroom, and tries to earn the man's sympathy by telling him that his wife died that week and it was his fault. But the man doesn't seem to care and furiously responds by saying (in a sarcastic, slightly skeptical tone of voice) "Well, I'm real sorry for your fucking loss, but your dead wife can suck my dick." An enraged Dexter attacks him, and after a brief fight, kills him. As he backs away from the man's cadaver, Dexter looks into the restroom mirror and sees Harry, who tells him that that's the first human thing he'd done since his wife's death. Harry tells him that he can show what hes feeling. Dexter finally lets his guard down and starts to shriek from both anger and grief over losing Rita. Harry tells him that he needs to go back. Dexter arrives at the funeral and while giving heartfelt eulogy, he realizes that Rita wasn't just part of a cover life-he actually did love her. http://dexter.wikia.com/wiki/Episode_501:_My_Bad

Edit:

I see Ankit Sharma found the name and his wiki page.