There is not much difference, except that "exchange" is used in more commercial and formal situations.
I exchanged the sweater I bought for one of a larger size because it didn't fit. -> I would not use "swap" in this sentence.
We exchanged places - > We swapped places. - the first is for more formal settings; the second very informal.
A "swap meet" is a market where people exchange things that they have and don't want anymore.
The stock exchange is where people buy and sell stock in an open market.
The "credit default swaps" is one market that caused all the bank troubles because it was informal and private.