Can't unset ability modifier of an attack's damage roll.

Plenty of magic and magic-like abilities don’t add ability modifiers to their damage. For example, I manually handle things like Paladin Divine Smite class feature. When I add damage dice for the radiant damage, the UI usually defaults the ability modifier to something and doesn’t let me unset it.