What version are you using and did you use Theme Builder?
In Washington DC and earlier, here's the simplest route: create one or more themes for your nonprod instance(s), get those themes all migrated to all instances (even production*, for preservation), and then, in each instance, manually set the property glide.ui.polaris.theme.custom = [sys_id of that instance's sys_ux_theme]. And of course, as you noted, preserve that property value so it doesn't get overwritten in clones.
*no harm in nonprod theme records being in production - users cannot select it
As I just found out now, in Xanadu/Yokohama - especially if you create the theme in Theme Builder - that property is not necessarily relevant. Theme Builder will create a m2m_app_theme record mapping the UX App "Unified Navigation app shell" to your theme and that will take precedence over the property. So what you need to do - in each instance - is : a) change the theme on that m2m_app_theme record to your nonprod theme or b) create a new m2m_app_theme record with a lower order (i.e. -10) with your nonprod theme. I would recommend b) so you don't lose your ability to switch to the prod theme in case you need to update it 🙂
And all that said, you will obviously then need to add this table to your preserve so it doesn't get overwritten.
Here's the documentation explaining more about this: Publish multiple themes in Next Experience
