Where do non-binary people fall for the gender specific awards in the Oscars?

Unknown, currently.

This isn't an issue the Academy has to address yet, and they've not published any guidance on it. As so few people identify as transgender/gender-queer, and more importantly, almost no major films have cast transgender/gender-queer people in high profile roles.

The statistics all show a tendency for white men to get the most Oscar worthy roles. There's currently a huge debate about how to cast more women and non-white actors/actresses in these high profile roles. Until that imbalance is addressed, I think it is unlikely transgender and gender-queer people will become prominent in them - so I suspect this is an issue the Academy will not need to address for quite some time.

There have been two transgender composers nominated. But as that entire category is gender neutral, there was no controversy there.

I would suspect that transgender people would be nominated based on the gender they identified as. Whether that is the gender they identified as at the time of the film or the gender they identify as now is unknown - but I'd suspect the more controversial the candidate, the less likely an elderly, white Academy selection committee is to nominate them.

For gender-queer, we have absolutely no idea. They may just choose a category and if the person rejects the category, ignore them. Of course, this is all just speculation - we just don't know.

A further point of interest is that there is a number of articles suggesting making the Oscars gender neutral. However, whilst many feminists are in favour of this, many others are not, as the evidence suggests the representation of women and non-white actors at the Oscars would simply diminish. Consider the gender neutral Best Director award. In the 85-year history of the Academy Awards, only 4 females have been nominated with just 1 winner - Kathryn Bigelow for The Hurt Locker.

So, whilst creating gender neutral Awards could help the Academy recognise a transgender or gender-queer person in future, it would also likely strongly increase the amount of white men winning the awards - so along with the positives, there are some very strong negatives to any move the Academy could potentially make here

A character in The Crying Game

is treated as a woman for the majority of the film, but was played by a man

When this actor

was nominated for Best Supporting Actor at the Academy Awards

many people considered it a spoiler.

So to answer your question, most awards shows will go by the gender of the actor, and if they feel a need, they will assign one. It is possible they will use the gender the person identifies as, but it has yet to happen, so who knows?

Update: not about the Oscar specifically, but for the Emmys, an article on Asia Kate Dillon says:

Dillon was surprised to learn that Academy rules say that “anyone can submit under either category for any reason,” says Dillon. “The Academy supports anyone’s choice to do that, and the Academy is not going to do any sort of check,” Dillon says. Indeed, Emmy rules for the acting categories do not specify gender qualification, saying simply “for a continuing performance in a regular series.”

Both Asia and their character identify as gender nonbinary. It's possible that the Oscars will take a similar approach.