How could Lt. Hicox fluently speak German and then give himself away by using the wrong gesture?

I'd say this gesture was something so subtle and unconcious that it just falls below any cultural assimilation. He has "counted to three" in the "British way" all over his life and even excessive exposure to German native speakers and culture probably won't change such a highly non-descript and intuitive gesture. You can speak fluently in a language and be acquainted with many of its cultural subtleties and might still not be aware of the most subtle an non-descript differences. It's not that if you do it the "wrong" way among a bunch of natives everybody will shout "hey, you're counting wrong". In most situations nobody will notice and thus nobody would have ever told him how to do it "properly". That's because in all but this particular covert operation nobody would have cared about how he counts to three.

Add to this that Hicox actually wasn't a professional spy trained for such situations. He was really only a British soldier who happened to have good knowledge about German movies and language (maybe even mainly from movies). It is true that they would have to face a situation amongst many German officers sooner or later (yet on a film fest, a métier Hicox would be much more familiar with). But in this particular situation they weren't prepared for any German soldiers at all, let alone such a cunning officer like Major Hellstrom, who continuously needled them and tried to bring them into revealing situations. Lt. Hicox just wasn't prepared well enough for this battle of wits and espionage where even the slightest misbehaviour counts (The situation may not even have gotten that far hadn't he spoken with such an Irish accent, which actually aroused Hellstrom's attention in the first place).

(As a counter example, I for myself think to speak English pretty fluently, even if maybe with a slight accent (though not stronger than Michael Fassbender's either), and have quite a bit of knowledge about Hollywood movies. This might even be sufficient to play an American film expert on a quest to bomb the Academy Awards. Yet I never ever heard about this difference in counting to three until this movie and would indeed intuitively order my drinks in the "German way" even when sitting in a bar full of U.S. officers.)

My wife and I both took German in high school and were taught by a native German. He told us the way that Germans count one with their thumb instead of with their fore finger. I always remembered this and she didn't.

While watching Inglourious Basterds I noticed the slip up right away she had no idea until after the shooting stopped and it was explained on screen. As Christian Rau it's one of those small cultural differences that aren't always noticed or commented on even when surrounded another culture.

What's stranger to me in that scene is the Nazi jumps automatically to these are enemy soldiers due to such a small cue.

The real explanation is much simpler (and less interesting) than some offered above.

You learn dactylonomy—finger-counting—when you're an infant, and you don't re-learn it as an adult unless there's a good reason to do so. It doesn't matter how thoroughly a British person immerses himself in a Continental culture, or vice versa, he/she is unlikely (and has no need) to switch dactylonomic codes, especially when the two systems are mutually intelligible.

Little Archie Hickox learned to count with his fingers on the knee of his Irish mother or father, who passed the art on, unchanged, from their own infancy, und so weiter, und so weiter.

As soon as you can do basic arithmetic, finger-counting becomes pretty irrelevant to your life, unless you order a lot of beers or work with a lot of trauma patients. You could theoretically go your whole adult life without ever counting on your fingers...

...until you have kids of your own. Then you enact the same, indelibly-rehearsed motor actions your parents instilled in you.

As a young man, Archibald Hickox' interest in German probably caused him to spend no end of time among no shortage of native speakers.

He could have spent 1000 years in the company of his echt German friends; it would have helped him shake off the Irish accent, to be sure, but it probably wouldn't have had any influence on his dactylonomic habits. After all:

1. he was hanging out with them to improve his German
2. he already knew how to count on his fingers
3. nobody at his Ye Olde Bavarian Beer Hall, be they echt Bavarians or just students dressed up in Lederhosen und so weiter, ever seemed to have difficulty understanding how many glasses he wanted when he signaled for zwei, drei, vier or fünf Gläser*. Even in Occupied France his gestures worked fine: note that, just before Hellstrom leveled a pistol at Hickox' Eier, the bartender delivered exactly 3 glasses to the table.

The ScheiBe hit the fan because Hickox violated what linguists call an ethonomathematical shibboleth.

Unfortunately Hickox was a film critic, not a professional spy. Any espionage instructor worth her salt would have drilled Hickox on the life-or-death importance of dactylonomic and other shibboleths, particularly because these are somewhat quicker and easier to master than a clean foreign accent.

But you have to know that you HAVE to master them. And we can't expect someone who's not a professional spy to have thought of these things.

I don't have any references to point to, but my original thought was that a large part of Lt. Hicox's German had come from watching German films. The gaps in his cultural knowledge were the result of his lack of exposure to native speakers.

Tarantino is a Director who is a homage maven. My thought is the slip up by Hickox is a nod to the Director John Sturges and his work in the 1963 WWII epoch "The Great Escape". If you recall, in that film, Bartlett and MacDonald evade capture after escaping from the Stalag. They slip away, but they are caught while boarding a bus after MacDonald blunders by replying in English to a suspicious Gestapo agent who wishes them "Good luck" in English. Similar to the blunder made by Hicox. Like in the game "Simon Sez"... They couldn't help themselves.

(While this is not an actual answer to the question, it does provide a relevant anecdote WRT the origins of this moment in the movie.)

According to Dutch director Martin Koolhoven in the Dutch news program De Nieuws BV on 11 August 2020: years ago he invited Enzo G. Castellari, the director of The Inglorious Bastards -- the Italian movie whose title Quentin Tarantino used as the inspiration for the title of his 2009 film -- to Holland.

Castellari was joined by his son (I assume this was Andrea Girolami, but Koolhoven did not mention his name) who helped his father with translations etc. Castellari's son told Koolhoven that he had been the one who had told Tarantino about this difference in counting to three.