The adverbs modify the verb. I'm not sure about exceptions.
The adverb modifies the verb. "Am going" may be present tense, but it refers to a future action. It's contextual. However, "I am going through menopause" refers to a present action.
Are we talking about wagering here?
Ah, the book!
I gave you the book. I'm sure you realize "you" is the indirect object of the verb. I presume it may be said to modify the verb, although I've never thought about it.
I confess wondering if "to you" is also considered a prepositional phrase.
But there's no question in my mind that "to you" describes the action "to give," and not the direct object "book." It answers the question, "Give how?" Not the question, "Give what?"