Is there a racial aspect to Max Cady's crusade?

During the 1991 version of Cape Fear Max Cady repeatedly insults his opponents as "white", as if to make a distinction between them and himself. This always confused me a little, since ethnically he is ostensibly white, too. I also failed to otherwise see a racial context in the film and its conflict.

Now I might just be seeing it too naively or superficially. There sure is more to being "white" than just skin colour or ethnicity, especially in a US setting. So maybe Cady is using it more in a class struggle sense of distinguishing himself from the "white" upper class that Bowden is part of and that allegedly thinks to be better than him. In which way is he genuinely seeing himself as non-white? Or has he just used "white trash" as a general insult without a necessarily racial context at all? Though, it would still seem there is something more to that.

Basically, is there any kind of racial aspect to Max Cady's crusade and by extension the film, be it real, symbolically on a broader level, or just imaginary as part of Cady's delusions? Or am I just making more out of it than there is to it?