How can one be nominated for an Oscar as "Best Actor" and "Best supporting Actor" in the same Film?

Can an actor be nominated for “Best Actor” and “Best supporting Actor” awards for the same performance?

No.

From the rules in their current state, under the acting awards (rule six) section:

4. The leading role and supporting role categories will be tabulated simultaneously. If any performance should receive votes in both categories, the achievement shall be placed only on the ballot in that category in which, during the tabulation process, it first receives the required number of votes to be nominated. In the event that the performance receives the numbers of votes required to be nominated in both categories simultaneously, the achievement shall be placed only on the ballot in that category in which it receives the greater percentage of the total votes.

How did Barry Fitzgerald get away with it?

Because this was back in 1945 (The 17^th Academy Awards). Rules have changed since then. According to Fitzgerald's Wikipedia article:

In 1945, Fitzgerald achieved a unique Academy Awards feat. For portraying Father Fitzgibbon in Leo McCarey's Going My Way (1944), he was nominated for both the Academy Award for Best Supporting Actor (which he ultimately won) and the Academy Award for Best Actor; voting rules were changed shortly after this occurrence to prevent further dual nominations for the same role.