Although not the Standard distribution, I agree, but given certain contexts, native North American speakers often omit "to". That is, "to" doesn't require overt realization in all instances:
Surface Structure: Whom did you give the letter (to)?
Underlying Structure: Whom did you give the letter...