Why isn't there Hogwarts without Hagrid?

He's saying that the school will be missing something for him if Hagrid is gone, that the experience is lesser without Hagrid. That it just doesn't feel like the same place without Hagrid there. But why?

Hagrid is a very important character to Harry. For several reasons:

• He told Harry that he was a wizard
• He was a friend of Harry's parents
• He's helped out in many situations
• etc

He is the one who originally told Harry about the magical world. This was a very important revelation for Harry. It quite literally changed everything he knew.
This will immediately cause a positive association with Hagrid - he's the one who brought into Harry's life a very important thing - magic.

He was also close with Harry's parents, which is again important. Harry's parents are important to him, and he's shown to feel that the friends of his parents are important to him.

Hagrid helps in many, many situations - such as with the Forbidden Forest, for example.

He's Harry's friend. He's someone that they can just go to talk to, when they need to - and the do. He's an important figure in Harry's school life. He's very firmly standing with Harry. He's almost a father to Harry, except more bumbling and stuff. Consider their reactions in The Prisoner of Azkaban:

"Look at him blubber! Have you ever seen anything quite as pathetic?” said Malfoy. “And he’s supposed to be our teacher!”
Harry and Ron both made furious moves toward Malfoy, but Hermione got there first — SMACK!
She had slapped Malfoy across the face with all the strength she could muster. Malfoy staggered. Harry, Ron, Crabbe, and Goyle stood flabbergasted as Hermione raised her hand again.
“Hermione!” said Ron weakly, and he tried to grab her hand as she swung it back.
“Get off, Ron!”
Hermione pulled out her wand. Malfoy stepped backward. Crabbe and Goyle looked at him for instructions, thoroughly bewildered.
“C’mon,” Malfoy muttered, and in a moment, all three of them had disappeared into the passageway to the dungeons.
_{Harry Potter and the Prisoner of Azkaban, chapter 15}

They're very protective of him, because he's their friend. They have almost as strong feelings about him as they do for each other, it's just a different type of relationship.

They like Hagrid so much that they're willing to go to his class and deal with Blast-Ended Skrewts for him. They don't want to take Care of Magical Creatures; they're doing it for him:

While they tucked into porridge and eggs and bacon, Harry and Ron told Hermione about their embarrassing conversation with Hagrid the previous evening.
“But he can’t really think we’d continue Care of Magical Creatures!” she said, looking distressed. “I mean, when has any of us expressed. . . you know . . . any enthusiasm?”
“That’s it, though, innit?” said Ron, swallowing an entire fried egg whole. “We were the ones who made the most effort in classes because we like Hagrid. But he thinks we liked the stupid subject. D’you reckon anyone’s going to go on to N.E.W.T.?” Neither Harry nor Hermione answered; there was no need. They knew perfectly well that nobody in their year would want to continue Care of Magical Creatures. They avoided Hagrid’s eye and returned his cheery wave only halfheartedly when he left the staff table ten minutes later.
_{Harry Potter and the Half-Blood Prince, chapter 9}

Hogwarts won't be the same because when someone who you like, and are used to having around, is suddenly not there, it feels empty. Lonely. Like something is missing. That's because there is something missing. Someone who you love is not there. And it doesn't feel good.
Harry is telling Hagrid that he is someone who, if gone, makes him feel as if something is missing. That he's important to him. That he cares about him. That he's such an important person to him that the entire school will feel different if Rubeus Hagrid is not there, fixing up the garden, and being his usual large, cheerful, self. It's a statement of how much he cares about Hagrid, and that they're standing with him.

It's just a figure of speech

Harry and Hagrid are friends, if you were at school and your best friend/favorite teacher didn't go that day. The school wouldn't be the same for you.

It just means that Hogwarts for Harry (and for anyone that likes Hagrid) won't be the same.

With Hagrid in Hogwarts, Harry has someone besides Hermione/Ron to talk to. We can see this throughout the books and movies, how Harry goes to Hagrid to talk to him about pretty much anything.

While Harry is the main character of the book series and movies and Dumbledore represents the height of wizarding excellence as well as the academic goals of Hogwarts, it's hard to argue that anyone else is the true heart of Hogwarts quite like Hagrid. He risks his life daily protecting the school, tending to creatures and would do anything for the students. That alone would grant him status in the workings of Hogwarts, however, when you put into context his history of rejection by the wizarding world (expelled in his third year after being framed by Riddle, his mother being a giant, and his general lack of social standing among wizards) his devotion to the school, its students and the creatures living among them makes him the heart of Hogwarts.