I think an important assumption in my previous comment is that your current alert design is already collecting the necessary measurements in the alert threshold conditions to make a determination between noise and signal. In other words, if your decision about which alerts were noise or not cannot in any way be derived from the features fed into your model, then the model won't be successful.