Only the special ability can be negated (or interrupted, or prevented). The numbers stay unless you discard the enhancement.
The reverse is also true - if you discard an enhancement (or any card for that matter) without negating it (or interrupting and preventing it), the effect is still activated/used (until the end of the current phase).
