Why didn't the USS Nathan James use air defense with SM-2 and ESSM in the opening scene?

While much of the Arctic Ocean is disputed amongst northern border countries, it was still considered international waters. They have been on this EMCON Alpha dual mission (run missile tests & escort science party) for four months with no contacts, friend or foe.

ENSIGN WILL MASON: Captain. I have three -- make that five -- unknown aircraft approaching, range 24 nautical miles, approximately 80 knots.
 _{source: The Last Ship - Pilot transcript}

This is where there is a bit of a problem with the timelines of the two locations; the science expeditionary party with their Naval Mountain Warfare Unit guards and the USS Nathan James. Three attack helicopters come over the horizon followed closely by two armed troop carrier helicopters. The three attack helos peel off toward the USS Nathan James while the two troop helos fly toward the scientists and their guards which they are pretty much right on top of.

At 80 knots (~90 mph or ~145 kph) it should have taken about twenty minutes for the attack helos to engage the ship at close quarters. Even if they immediately doubled their airspeed to 160 knots after leaving the other two helos that would be roughly ten minutes. The firefight with the science party was going to be over in a lot less than that and the NMWU guards should have had plenty of time to radio the Nathan James with an account of the unprovoked attack. Perhaps the original novel by William Brinkley makes more sense of this.

So at this point, we have to suspend disbelief a bit and assume that the two areas of engagement happened at roughly the same time with the fight at the Nathan James only slightly behind the attack on the science party. If you cannot do that, there is no point in reading any further.

If you're still reading then the two timelines are roughly synchronized. As the attack helicopters approach the Nathan James, General Quarters is sounded and the TAO (Tactical Action Officer) is apprised of the incoming aircraft. As the CIC (Combat Information Center) is apparently unaware of any hostile actions, the order is given to hold fire. Evasive action is initiated and radio contact with the incoming aircraft is attempted.

[ Radio chatter ]

ENSIGN WILL MASON: ... United States naval warship. Identify yourself and alter your course, or you will be subject to defensive actions. Over.
 _{source: The Last Ship - Pilot transcript}

That's when the Russian attack helos start strafing and firing missiles at the ship. After the first pass, the Nathan James' crew deploy their countermeasure chaff and Captain Thomas Chandler gives the order.

CAPTAIN TOM CHANDLER: Take 'em out.
 _{source: The Last Ship - Pilot transcript}

On the attack helos' second pass two additional Russian missiles are neutralized by chaff while a third strikes the ship. One helo is shot down with the 25mm autocannon controlled remotely by the TAO, another with the Phalanx CIWS and the last with the 5-inch Mark 45 Naval Gun.

            _{5-inch/62 caliber Mark 45 Naval Gun}

Ultimately, the Captain and crew of the USS Nathan James got the bad guys, saved the scientists and got a clue from one of the Russian soldiers before he died all the while not double-tapping missiles at approaching helicopters that haven't (to the CIC's knowledge) shown overt aggression.

While the Nathan James' crew was not strictly acting under a 'do not fire until fired upon' rule of engagement, audiences are more likely to decorate protagonists with white, 'good guy' hats when they aren't the aggressors. This battle scene sets the tone for the remainder of the series; Captn. Chandler and his crew are the altruistic 'good guys' that are risking their lives to save the world.

Is that somewhat realistic?

In fact, the whole series is ripe with inaccuracies. I remember laughing out loud when the Navy Tech began beating the handle of the fuel valve with a wrench at Guantanamo.