Can you hurt upside down faceless monsters without magic at all?

Hurt? Yes. Kill? Unclear.

Both the (presumably) fully grown demogorgon in the first season and the "demo-dogs" in the second season appeared to be physically harmed by things other than Eleven's telekinetic powers (what the question calls "magic").

In season one, Nancy and Jonathan seem to be able to hurt the Demogorgon using fire and other weapons, forcing it to retreat to the Upside Down. In season two, the demo-dogs are shown to be affected by gunshots, especially at the end of the last episode when Eleven is attempting to close the gate.

However, in all cases they appear to be highly resilient (a few gun shots aren't sufficient to do serious damage), and may even have rapid regeneration abilities (in or outside of the Upside Down). We also don't know if it's possible to fatally wound any of them as that's never been explicitly shown.

Although I think the story seems to be set up with a kind of X-Men-like narrative, where it seems like the metaphysical characters are a fundamental part of dealing with the metaphysical creatures of the Upside Down, I would argue that both the human characters without abilities are still important and that we have seen some non metaphysical ideas play a role.

1. If it wouldn't have been for the kinetics between Will Byers and The Shadow Monster, we know that fire (Heat and Light) harms The Shadow Monster and may have destroyed it. It's unclear though if the same goes for the Demogorgan, because we have never seen that happen and/or know if the two are at all related.

2. Dustin's bonding with the demi-dog (a similar creature like the Demogorgan) also suggests that some of the creatures from the Upside Down can, in some cases, imprint with human characters and form meaningful relationships. If it wouldn't have been for Dart's early interactions with Dustin, Dustin and friends may not have escaped the tunnels.

3. Many of the characters did things from driving around, to teaming-up to find solutions and there is an argument that even the metaphysical characters rely on the non metaphysical characters and it isn't just the other way, because the story is character driven.

Yes, and most of the time they are killed through physical means. Although highly resistant to the average gunshot, demodogs in season 2 are quite clearly vulnerable to blunt trauma. The "magic" you're referring to, I assume, is Eleven using her powers, but here's the thing, it's not her powers that killed it, it was the force of impact after she swung it around and threw it through a window. It might seem like it's the "magic" doing the work, but take anything capable of hitting them with similar force, and it will kill them just like El killed that one (or possibly more than one that we didn't see). Resistant at times, they can be killed by gunshots, such as when hopper took a stand while trying to defend El as she was closing the gate.

Quite frankly, the only "magic" that killed them throughout the show, was when the gate was sealed and they started dying off due to their connection to the upside down being cut off. As for the demogorgon, although even more resistant to bullets (seemingly bulletproof), it still is vulnerable to blunt force (Steve's bat), and it's also highly vulnerable to burning/fire, which is possibly even lethal if exposed to for long enough. El's abilities may have killed the demogorgon which may qualify as a supernatural death, but it's not exactly confirmed since they both disintegrate and disappear in the exact same way. Only one is shown to re-materialize, and that's Eleven.

Last but not least, even the mind flayer is shown to be susceptible to blunt trauma, again when El swung it around, and so quite possibly it's once again vulnerable to fire (fireworks). Though, I wouldn't count on whether it can be killed that way or not.