Does anyone else think that there's a difference in placement based on whether or not you actually DID do it? Or at least, an implication of that? You can make sentences that mean each of the following using a different word order and vocal emphasis, but this is my first interpretation:
I never would have: If I had known you were returning from your trip on Thursday, I never would have agreed to have dinner with Marge on Friday. (But I did agree to do so.)
I would never have: You thought I was having dinner with her? I would never have agreed to have dinner with Marge! (I didn't and I wouldn't.)