Sauce is right. Since Magic Charms is not a weapon, it doesn't follow the same rules as a weapon. It follows the rules of an Artifact, which remain under the control of their owner, even if a card that's holding them ends up under another players control.
It might make more sense if you think of it in terms of placed cards, like Destructive Sin. If I place DS on your Hero, I still control it. In a MP game, if a 3rd player bands to your Hero, they don't control DS, I still do.
Magic Charms is not a place ability but it functions in a similar way for this example.