...this has got me thinking of where /x/ contrasts with /h/ vs /k/ in English as people actually pronounce things; language of origin/orthography definitely matter but I think it's mostly beginning of the word people are most likely to interchange with /h/, end of the word with /k/
Yeah, orthography no doubt matters. E.g., no-one says “Khan” as “han”.
Greek X in general usually gets pronounced as 'k'/"kai" in US English, I assume for reasons of how schools that taught Classical Greek went about doing so for the past century or two. It's particularly noticeable in college fraternity names.
And people who are use /x/ in other contexts don't usually apply it to Greek loanwords unless they're making a point about speaking Greek