AvangiI'd say "speak" is a "bare infinitive" and "English" is it's direct object.
Me too. I don't know what else it could be.
However, the adjective idea is weird. 
father is the subject of the infinitive. It's the father who does the speaking of English, after all.
You have interlocking clauses here. I listen to my father. My father speaks English. The second becomes subordinate to the first, and its verb becomes non-finite.
CJ