He's old, out of shape, and half machine. Starting with the battle between Vader and Obi Wan in Episode 4 (which was far worse in quality between two fully trained Jedi), Lucas says:

In Episode 4, it was a fight between a very old man, and a man who was only partially a man, mostly a mechanical being, so it wasn't really much of a fight at all.

He goes on to describe each of the major one on one fights in the series, including the differences in attitude Luke has going into the battles in Episode 5 vs Episode 6. Episode 5 is really driven by revenge, where Episode 6 he sees the big picture and is doing what needs to be done.

TL; DR. The OP gives Vader too much credit, and Luke not enough.

There is an ongoing drama of Vader plotting against Palpatine from the get-go, but it seems that in order for Vader to survive, he must remain subservient to Palpatine unless it's highly likely that Palpatine can be overthrown.

I'm completely unaware of this ongoing drama that's being referred to. AFAIK, the closest Vader comes to saying anything remotely like this is when he and Luke fight on Cloud City in Empire Strikes Back:

VADER: You can destroy the Emperor, he has foreseen this; it is your destiny. Join me, and together we can rule the galaxy as father and son.

If Vader's telling the truth in that the Emperor foresaw it, and that it's apart of their plan, then this wouldn't necessarily be considered plotting against the Emperor (since the Emperor was the very one who came up with the plan).

During the lightsaber duel with Luke in the throne room in Return of the Jedi, Vader is basically a push over. It's clear from rewatching the scene that Vader is either a push over or has completely given up.

Remember that, between films V and VI there was a lapse of about a year (it's never officially been established, but this is the most widely-believed amount of time). During this time, Luke continued training in the ways of the Force, and got stronger both physically and spiritually. Vader, however, only got weaker, and more dependent upon his machinery.

Just before Vader took Luke to see the Emperor, while looking at Luke's crafted lightsaber Vader says,

VADER: You've constructed a new lightsaber. Your skills are complete. Indeed you are powerful as the Emperor has foreseen.

This supports the idea that Luke had continued his training and gotten stronger. Vader, on the other hand, had at that point become more accustom to hiring bounty hunters to do his "dirty work", as was shown in Episode V. And so, perhaps over the years Vader was getting less and less combat experience, given his poor health (end of Episode VI); or, when he does get into combat, he uses the Force to compensate for his saber (Episode V duel).

With this in mind, what may seem like Vader being a "pushover" could just be a result of Luke's significant increase in skill and strength since the last time they fought, making Vader's diminished skills that much more obvious.

(side note: Luke was ~24 y.o. and Vader was ~45 y.o. during the throne room duel.)

At the same time, Luke isn't exactly good either; he's fine against blaster-armed enemies, but he's really a complete novice in a lightsaber duel, whereas Vader has killed countless fully-trained Jedi masters during and after episode III.

First and foremost, Episode III happened over 20 years before Luke and Vader's showdown in the throne room. IMO, I don't think the events in Episode III can necessarily support any arguments regarding the throne room duel (within the context of the OP).

If we're to still consider Episode III though, the only Jedi [equivalent] that Anakin kills is count Dooku. Aside from that, Anakin doesn't kill any Jedi masters. Instead, all that's shown is him: (1) slicing off Mace Windu's arm; (2) slaughtering a bunch of padawans; and (3) killing Nute Gunray's group on Mustafar.

When Anakin fights Obi-wan though, he loses miserably.

Again, I'm not sure the events from Episode III should be considered, however, even if they are, there doesn't seem to be much evidence to support Darth Vader being as skilled of a swordsman as the OP suggests he is. Also, AFAIK, most all the Jedi were killed by clones during Order 66, and not Vader.

The final duel in the throne room seems to indicate that Vader wanted somehow to betray Palpatine, so perhaps this is why he went easy on Luke. However, if Luke agreed to join the dark side, would Vader and Luke really be successful where Vader would not have been?

After watching the throne room duel again, I'm not seeing any indication of Vader wanting to betray the Emperor, except until the Emperor begins attacking Luke.

Vader did though, to some degree, seem to not be as aggressive as he had been during their last encounter (i.e., he doesn't use the Force to push/throw, etc.), but he was still exercising Luke's abilities (example).

If there was any hesitation on Vader's end, it was a result of the conflict within Vader to not kill his own son, just as Luke mentions several times throughout the movie.

As for the last question, "if Luke agreed to join the dark side, would Vader and Luke really be successful where Vader would not have been?", I'm not sure what's being asked here, but if the OP can clarify then I'll attempt a response.

Because, to quote Darth Vader: You underestimate the power of the dark side.

Luke is powered by hate and anger. Feelings that Vader doesn't have towards Luke or doesn't have at all. As Palpatine says to Luke "your hate made you powerful".

The moment when Darth looks at the emperor torturing Luke is not only the moment when he turns back to the light side, it's also the moment when he regains his feelings. As you mentioned, he is a machine. And machines don't have feelings.

The following may only be marginally helpful answer, since it takes place during the post Revenge of the Sith time period, but it may provide some insight or proof that Darth Vader still struggled with his feelings and may prompt questions about the nature of struggling in general.

Darth Vader: The Chosen One Comic - ongoing (2017)

The series litarally picks up from when Anakin awakes in his new armor. He quickly gets into it with Palpatine, not struggling to fight back. When Palpatine realizes that Aanikan/Vader no longer has his previous lightsaber, Palpatine pretty much asks him, if he wants to live or die. Anakin/Vader chooses to live, but this sets up a series of quests for Vader to acomplish, including hunting down a remaining powerful Jedi, killig him, and then taking his lightsaber so he can cause the kyber crystal trauma, make it bleed, and turn it red--the color of his rage.

From Comic writer Charles Soule

Charles Soule: In my head, I call this book “Vader: Year One.” While the Sith Lord does exist prior to the start of this book, during the final act of Revenge of the Sith, I think this is where his story really begins — at least in the form most people think of when they visualize Darth Vader. As far as what’s going on in his head, here’s my description of his mindset from the original series pitch: “Vader, on some level, knows that he is a monster — he has to know this, because he still has that famous spark of good left somewhere deep inside him.” He knows (again, probably unconsciously) that the moment he allows himself to even consider a path that isn’t complete and utter darkness, he’ll fall — and he’s right, as we see in Return of the Jedi. So, he doubles down on the dark side as almost a coping method.

Charles Soule: I was actually a bit nervous about it at the start, since as you point out, Vader is low here. There’s a lot of death, a lot of destruction in the series — that’s all Vader really wants, and I knew I’d be writing a book that didn’t (almost couldn’t) shy away from those elements. I’ve found it surprisingly cathartic, though. Vader is pure emotion, pure rage, and it’s a way for me to work that out a little without actually turning to the dark side myself. Who knew? Vader might not be having much fun here, but I definitely am, and I think the readers will, too. The series is very metal, if that doesn’t sound too bizarre (or too on the nose, considering the main character is more machine than man). http://www.starwars.com/news/in-marvels-new-darth-vader-series-we-will-see-the-sith-lords-rise-the-construction-of-his-lightsaber-and-more.

The reason I think this is important, is because it does prove that even as Vader, he has emotions and that the sides of the Force seem to rely on what emotions one fixates on. In addition the idea that Palapatine basically puts Vader through the ringer to feel more pain, presumably for him to more heavily embrace it, could be seen as a transtional process, meaning that there is still a "struggle" for Vader (at this time). However shortly hereafter his transitions seems more perminenet (until Luke) according to a Star Wars Rebel's producer when discussing Vader's interactions with his former apprentice, Ashoka Tano.

Did Ahsoka’s battle with Darth Vader have any lasting impact on that him?

“I personally have never felt that anything changes Vader until Luke,” Filoni said. “The Vader that we encounter in Rebels was always meant to be the one devoid of emotions, except for anger, hate and suffering. That he was so trapped inside himself because of the terrible things that happened. Anakin never thinks of himself as betraying his friends. He sees it as his friends betrayed him and the Republic. He has to live on that side of the fence because the truth is just too damming.

“So he wants to destroy Ahsoka because she represents his past. She represents knowledge of who he was and he wants to wipe that out. His son represents a potential future because his son wouldn’t know who he was. So he could build a new galaxy together with his son. His apprentice is his past and he needs to destroy her.” https://io9.gizmodo.com/the-producer-of-star-wars-rebels-answers-your-burning-q-1768048083

And from Wookiepedia

. On the Death Star, an enraged Palpatine declares that if Luke will not turn to the dark side, he will be destroyed, and uses Force lightning against the young Jedi. He slowly increases the intensity of the lightning,torturing Luke. He then calls out to his father to help him. But as the Emperor prepares to deliver the killing bolts, Vader must make a choice, he looks at Luke and then the Emperor, conflicted whether to save his son or to continue serving his vile Master (just like when Anakin had to choose between Palpatine or the Jedi Order in Revenge of the Sith).

Moved by his son's cries for help, Vader makes his decision — he cannot let one of the two last remaining links to his deceased wife, Padmé Amidala, die. With that, he lifts the Emperor into the air and throws him into the Death Star's power core/ reactor shaft. The Emperor disappears into the abyss, screaming, and then he is gone. There is an explosion but during the process of killing Palpatine, his lightning enters Vader's organic remains, striking Vader's life support system and his respirator (Vader's mechanical lungs), mortally wounding him. The Millennium Falcon and its remaining Rebel fighters enter the bowels of the Death Star, and some fighters engage in a point-blank attack on the Super Star Destroyer, causing the Imperial flagship's destruction.

"It is the name of your true self. You've only forgotten. I know there is good in you." -Luke

So to answer your question, I think it is very clear that Vader wasn't so much weak, as Luke just happened to be thing that allowed him to change--hope for something better--for himself and his son. He was able to feel the good inside him, because of Luke and that was enough for him to finally surcomb to his better half, and overthrow Sidious, but that he didn't fully give into it, until that moment where Sidious is hurting Luke.

(Also it's not like he didn't cut of Luke's hand and let him fall into to the abyss in Empire Strikes Back. I think he was pretty efficiant. And he had been suffiantly dueling others between ANH and ESB as seen in the other Darth Vader comic.)

Anakin was a tremendous natural talent - either naturally born of the Force or one of Darth Plageius' cast away experiments to create life (see 'Darth Plageius'). It is reasonable to believe Luke had as much raw ability as his father.

In the expanded universe (and in the movie) much of what Jedi do to win a fight happens outside the camera eye - identifying and navigating the future. We saw an example of that when Anakin jumps off his speeder at just the right time to land near Padme's would-be assassin. Darth Sidious, a prodigal Sith, was watching the fight - we should presume his approval meant that, to his Sith senses, both combatants were fully committed.

Vader goes through a pretty big emotional shock during the fight - and before. Anakin's built his self image as a protector of his family - avenging his mother, selling his soul for his wife, punishing Padme's imagined infidelity with Obi Wan, and avenging what he believed was Obi Wan's betrayal to the detriment of his wife and children. He has long thought both his children dead - the result of Obi Wan's trickery. He believes Sidious when he's told the children are dead. Then he finds his son, undermining his faith in his master. Then, during the fight, he discovers his daughter is alive and well. Vader and the Emperor were both useless in the one area where Anakin had needed them most. And when Luke roars out of hiding to defend Leia, Anakin is not the protector of his family - he's the villain. For a moment, Vader's whole excuse for existence has collapsed. Anakin is emotionally helpless. And since the Force is partly about connecting to emotions, he may have been helpless in the Force also.

Also, Yoda may have been training Luke for little but this fight. The little green sage must have known it was coming, and had spent most of Anakin's life watching the Jedi mature. While Yoda may not have predicted Anakin's betrayal, he might have been able to train Luke to the weaknesses in Vader's technique.

As to how Darth Sidious pretty much took Vader apart - Vader may have literally been Sidious' co-creation with Plagueius. Metaphorically, Sidious was grooming Anakin emotionally since Vader's pre-teen years. Since feelings relate to capability with the Force, Sidious had a roadmap to dismantling his pupil. Also, we don't know the degree to which Sidious was using the Force to keep his pupil alive beyond what the technology of the time could do. And, Sidious may have put technical safeguards against betrayal in Vader's life support equipment.

Alternatively, the fight could have all been theater. Vader has already unveiled his plan to Luke that they topple the Emperor and rule together. Vader genuinely believes in the government he and Palpatine built together - keeping the gangs of bullies and the corrupt in check by being the strongest bully on the block. Vader may have hidden his feelings of betrayal from his master under his genuine belief in the mission. In the Timothy Zahn books, the Emperor is partly distracted during the fight because the Emperor is personally conducting his troops, using the Force to coordinate their actions supernaturally. So, it's possible the Emperor was too distracted to notice his apprentice wasn't giving the fight his all.

In this case, where did Vader's plan fail? Probably when Luke refused to kill the Emperor - even at the cost of his own life. Vader seemed to believe himself incapable of besting his master, and probably felt that Luke must be the one to do it. He probably did not imagine Luke being so committed to his virtues to die for them, Anakin wouldn't have.