Just change it into "I must go" - then it's easier to analyze:
"go" here is the full verb that is in its infinitive and follows the modal auxiliary must - without to!
I have to go:
"have to" is the substitute form for "must", "go" again is the full verb in its infinitive.
"Have to" here is a...