Since SoG is "negate and discard", negating the Wanderer/Exchanger undoes the exchange and returns it to the starting LOB. If that's your LOB, it's not longer a valid target for the rescue, since Lost Souls can only be targeted for rescue in an opponent's LOB.