Oh God , this calls for a study!
Yes, "with" and "by" changes the meaning but don't want me to make a generalisation. In fact, if you look this word up from the dictionary, you will find the answer.
1- The group was smitten with terror BUT he was smitten by her charm. You see that...