You have to talk English with him.
With specific languages, the more idiomatic word is
speak, so it's not very often that you will hear
talk English, talk Spanish, talk Chinese. These are normally said as
speak English, speak Spanish, speak Chinese.
CJ