Whose baby was it?

The film never revealed who the father is. I think its Costigan.

From yahoo answers by jeanette:

They never revealed who the father is. So you can take a guess. My theory is that it was Costigan. Since she didn't know the real situation, I think she was just going to stay with Sullivan and raise the baby as his. He was stable and secure, so she thought. Later when she finds out the real truth, you can see she's clearly distraught at the funeral. She walks away from Sullivan like the baby is no longer a concern of his. That's just me.

From Robert Love at quora.

Infertility and the relationship between fathers & sons is a theme in The Departed.

Early in the film, Staff Sergeant Colin Sullivan (Damon) is confronted by his girlfriend, Dr. Madolyn Madden (Farmiga), over a failure to perform sexually the night before: "So, do you want to talk about last night?" she asks, peeling a banana (not subtle imagery there). "You know, it's all right. Guys tend to make a big deal out of it. It's actually quite common."

Later, when Madden tells Sullivan that she is pregnant, she doesn't reveal the identity of the father. The allusion is that the father is Billy Costigan (DiCaprio), with whom she had an affair, and not Sullivan, who is impotent.

We can be fairly certain that Billy Costigan (DiCaprio) is the father, not because Sullivan couldn't get an erection one time, but because there is a recurring theme of infertility in the evil characters in the film.

There is the scene referenced above (Robert Love) where it is mentioned that Collin could not perform sexually.

There is a scene where Gwen (Frank's wife) is seen in their apartment reading 'Getting Pregnant.' Suggesting that she's having trouble doing so, perhaps because of Frank's infertility.

Toward the end, there is a somewhat forced exchange between Frank and Colin, where Colin implies that Frank couldn't produce children:

COSTELLO
I know you, Colin. You know I’d
never give you up. You’re like...
COLIN
A son...to you? Is that what it is
about, all that murderin’ and
fuckin’ and no sons? What are you,
shooting blanks?

On the other side are the 'good guys.'

Captain Queenan is proud of his son and mentions him multiple times. His last line in the film is, 'my boy's studying law at Notre Dame.'

And in this camp, fits Billy--one of the good guys. That said, I think we are to conclude that the baby is his.

Everyone seems to forget when Bill (DICaprio) gave her the copy of the same tape he sent to Colin (Damin), she (Madolyn) wanted to tell him something but hesitated and he told her to wait 2 weeks. Also after Madolyn heard the tape she stormed into the room saying "and I thought I was the liar" she had already given Colin the ultrasound picture. So, it had to be Costogan's (my opinion).

In the scene where Billy gives Madolyn the envelope in case of his demise, Madolyn wanted to tell him something but Billy told her to think on it hard and tell him again in two weeks. There are other reasons why we should believe Billy is the father; the theme of infertility in the bad guys (Frank) vs fertility in the good (Queenan) as well as Colin’s impotence, but this is the best explanation and quite convincing. She was going to tell Billy something important. There is nothing else that could be so important to the plot or emotionally impactful as the fact that she was going to tell Billy he was a father. Add to that my personal belief that it was a way to soften the blow to the audience of Billy‘s death, because he turned out to be a good guy and after everyone else is gone he’s the only one given a legacy.