You need a script include with script to check if record exists in the related list by querying the related table.
Then Create your UI action and in the condition, call the above script to check if related list is empty.
Please mark this response as correct or helpful if it assisted you with your question.
