I disagree, “should’ve” and “should of” sound virtually identical when spoken (at least in some regions, can’t speak for all pronunciations). I can imagine why a non english native speaker would have trouble with this, though I’m not disagreeing with it being a common issue amongst native speakers as well.
This is how I’ve always understood it as well. The two spellings are homophones so it’s a pretty easy mistake to make.