In the relative clause it's actually the relative pronoun
that that you're interested in, right?
all that I [have / had] to do:
that is the direct object of
do, in the most straight forward reading.
(that is assumed if omitted, of course.) But there is ambiguity, and
that could also be the direct object of
have (or
had).
I have to do these things. I have these things to do.
Still, even in the second sentence, the object of
have is simultaneously the object of
do. Paraphrasing with bad grammar:
*I have these things such that I must do these things.
CJ