Why did Ji-un not realize Hwi's secret?

In The King's Affection, Ji-un seemingly does not realize that Yi Hwi is a woman

until she reveals her secret to him.

and even after this, had to be told (!!!) some time later that

she was Dam-i

However,

• Ji-un had known Hwi when they were both children, so he has some idea of what she looked like.

• Near the very beginning of the series, he saw Hwi without her typical royal garments and recognized her as a woman (thinking she was a court maid).

• When he next met her (as the Crown Prince), he seemed to recognize her from this encounter and seemed somewhat confused.

• Hwi had the book that Ji-un had given to Dam-i.

• Hwi told him a highly specific story about rescuing a friend from drowning in a particular pond when she was a child, which is the exact thing that happened between Dam-i and Ji-un. Ji-un even notes that only Dam-i and he had known about the location of that particular pond when he first sees her beside this pond.

However Ji-un seems completely surprised, and even says outright that he was sorry for not realizing who she was, explicitly confirming that he did not know it before. The court lady and eunuch who were Hwi's only confidants were justifiably so concerned about the possibility of someone realizing her secret that they insisted everyone remain five steps away from her. As such, how could Ji-un, with all the clues available to him, fail to realize that (a) Hwi was a woman and (b) that she was Dam-i?