If you're using counters equal to the turns set-aside, you would get the card(s) back when you remove the last counter. Think of them like locks, once the final lock is removed you can access them again.
Set aside counts are updated during the prep phase of the card's owner, so the first counter would get removed during your opponent's next turn and the second during your opponent's turn after that.