visual studio code – Disable color decorators for all files except CSS

When I write a hex color code in a CSS file, a square appears to the left, filled with the hex code’s color. For example, if I type “#ff0000”, a red square appears to the left. This feature is quite useful while writing CSS, but it’s an annoyance in other languages (where “#” might have a different meaning).

This answer says you can use "editor.colorDecorators": false to disable the feature for all files. However, I want to preserve the feature for CSS files (i.e., where it’s actually useful), and disable it for all other files (i.e., where it’s useless). How can I do this?