Why was Medivh hostile towards Khadgar?

It is difficult to pinpoint Medivh's motivations in his first encounter with Khadgar, as shown in the movie. We can speculate, but it is important to note that we are dealing with key plot points that differ in the video game series. We will look at each of your points, in regards to why Medivh may act in this way.

Medivh is aggressive towards Khadgar

Medivh attacks Khadgar with excessive force

It is important to take note of Medivh's state of mind, at this point. We later find out that Medivh was the one to bring the Orcs in from Draenor, but the movie does not truely emphasise why. Long story short, Medivh's mother fought against the penultimate evil, an old god by the name of Sargeras. This is the guy that created The Burning Legion¹, in the first place. As his last act, Sargeras posessed the unborn Medivh, who was still in his mother's womb. At the point at which we are introduced to Medivh, he has all but lost his internal battle for control.

Khadgar may be a friend, but it is difficult to establish Medivh's exact frame of mind, at this point. As mentioned, he likely fears that Khadgar was snooping, and is afraid of being discovered. It is also entirely likely that he is hostile towards Khadgar, after whatever falling out that led to Khadgar renouncing his vow; as far as I am aware, this is a key plot point that does not happen in the video game series. As far as I am aware, the video game plot has Khadgar remain as the loyal apprentice to Medivh, right up until he is forced to kill his master.

Medivh acts as if Khadgar is his successor

Medivh acts as if Khadgar is his successor ("...what you're going to do with the place once it's yours?", also, later in the movie says something in the lines of "You are not the Guardian yet")

Considering Medivh's state of mind, as previously mentioned, I boil this down to general confusion. Perhaps Medivh has seen Khadgar after an extended period of time, remembers the anger associated with him leaving, but fails to recall that he actually did leave.

On the other hand, perhaps he interprets that Khadgar left to become stronger than him, and has returned to take over his command. This form of paranoia and/or expected behaviour is especially understandable when we consider that Medihv is currently posessed by the creator of The Burning Legion¹.

Khadgar does not want to be in Karazhan

Khadgar isn't supposed to be in Karazhan ("I didn't want to come here...I swear...Guardian")

I do not necessarily think that he is not suppose to be in Karazhan; I simply think he is extremely hesitant to return. We are not told what went down, when Khadgar left, but we can assume from his hesitancy to return that it was not the sort of thing that invites a warm welcome. Again, this particular plot element is unique to the movie; so we do not really have as much backstory to draw off, in comparison to various other elements².

It is also worth noting that it is fairly clear that he connects his previous leaving with the hostile welcome he receives; however, as previously suggested, Medivh's motivations may easily have been unrelated.

Concerning extended lore

As previously stated, in another answer, there is plenty of lore to read up on concerning these two characters. However, a word to the wise: Warcraft is an expansive universe. There is no guarantee that any game lore you read about Medihv, Khadgar - or for that matter, anyone or anything else - will hold true, in terms of the movie canon.

I would still highly recommend further reading. Blizzard write some good fiction.

_{¹ I do not recall if The Burning Legion are actually brought up, in the movie; however, they are the armies of demonic forces that seek to purge all life from the world. For what little they may have been brought up, expect to hear of them in far greater detail, in any potential sequel.² On that note, if you enjoyed the movie, check out the video games and their lore. The movie misses out on a great deal of underplot, both before, after, and during the events within the main plot; and it is no surprise. Warcraft is a massive IP. It would be somewhat like watching the Lord of the Rings movies, and expecting the same detail as the books.}