If English-speakers learnt how to pronounce the Latin expression per se, they wouldn't want to transcribe it as per say. The e of se is not a diphthong. In no English word is a solitary e pronounced as a diphthong, so why imagine it is so in other languages? Try removing the t from set.