First, try to forcefully set the full RGB mode.
Second, 10-bit is only supported for their pro-grade cards, no wonder you see the black screen, even if the monitor supports it.
Third, newer cards are tend to be overbright, so again no wonder there's a drastic distortion, they simply distort the vibrancy.
Try lowering the digital vibrancy setting.