Ah, my apologies. I misspoke. While keeping memories from idols in general is a method of control, it makes less sense to not have them remember people from their own stars. That adds an emotional component to these games where they force choices upon us, so it's no surprise that your memories would be a laundry list of those present from Teyvat.