Retire your vulnerability quarterback

[myke2]

        Welcome to the first of what will hopefully be many posts on security topics impacting organizations of all types. Those who know me know that I spend much of my time on the road speaking to a large network of CISOs, CIOs, and Security & Enterprise Architects, as well as analysts of all backgrounds. I'm very fortunate that many of the folks I speak with are open about their largest challenges. One of the more resounding pains that I hear pertains to patching and vulnerabilities, which isn't much of a surprise if you read or watch the news.

Your vulnerability quarterback is unlikely to be voted MVP

With an overwhelming number of patches and vulnerabilities to deal with, handling them requires a game plan, usually run by your vulnerability quarterback. Who is this key player? It's generally someone with significant security and IT experience who determines what needs patching and when and ensures patches are tested before going to change management. Unfortunately, the quarterback is also often the bad guy, forcing user updates, interrupting services, or outright "breaking" things. It's a thankless job.

But your vulnerability quarterback has more problems than grumbling peers and end-users. Data used for these patch decisions often comes directly from your vulnerability management tool that is well-informed about vulnerabilities but not your organization. Vulnerable items are mapped by the IP address, making it hard to tell what's affected without manually matching IPs to to applications that require patching. Moreover, the people handling the patching may not be responsible for the underlying system.

Now, you might ask if it would be easier to just have IT log into your vulnerability management tool. But good luck having IT log into a security point solution, it would be like asking your toddler to make an appointment for their next round of shots.

So if that doesn't work, how about getting everyone together for a meeting to figure out how to proceed? Now everyone is spending an hour talking about the work instead of actually doing it. The first half is spent arguing over which version of the vulnerability export report they're supposed to discuss, and the second half involves complaints from people who are upset that they had outdated info. Little is accomplished beyond wasting time that could have been used to fix other urgent issues.

Speaking of urgent, how do these teams cope with high-profile vulnerabilities like POODLE, Heartbleed, or Shellshock? When these vulnerabilities are exposed, there's generally a panic as your security team tries to determine what specific items in the organization are vulnerable. In many cases they are treated like Security Incident. The quarterback is put on the spot when asked if everything is secure, and he's especially hampered by the difficulty in getting relevant information out of the vulnerability management tool.  

There's a variety of response solutions…and many of them are bad

Now that we've discussed the problems facing vulnerability response program, how do we solve them? Let's start with a few ways not to solve them:

Creating tickets in your IT service management system. While it sounds reasonable, there are some problems with this idea. The team creating the tickets isn't the same one who will be actually doing the work. What happens if one of the patches fails? Do you reopen the ticket or create a new one? If/when you open a new ticket, the SLA and any metrics will likely be lost. The IT system wasn't designed for this type of work and isn't capable of reporting progress. As I'm sure many of you know patching is a fact of life in IT, and evidence that the program is working is difficult today. In addition, if all you have is an IP address for each vulnerable item, how do you determine which team is assigned to the ticket? How do you know the security runbook is being followed, and how do you handle approvals for exceptions or emergency changes?

Reports for everyone. You could create a giant Excel spreadsheet for everyone to work from so they all have access to the latest data, but tracking will become a nightmare. The next thing you know, the spreadsheet is corrupted, and you hope the last backup isn't too out of date. Or you could have a few smaller reports: one for each network or one for each team. But now someone has to go and update multiple spreadsheets for each new vulnerability. "Who has the latest version?".

Email. Do I need to explain how quickly this will become a huge mess? Not only is it hard to track, but now you're looking at hundreds of thousands of additional emails clogging people's inboxes.

So what's the real solution?

A lasting solution needs to address the big problems: getting the necessary information to the right people so they can address the most critical vulnerabilities efficiently using a method that can be easily tracked from discovery to remediation.  

The first step involves automatic routing of vulnerabilities to the right people the first time rather than rerouting with each new event. Using change ticket templates and macros makes the process not only faster but also consistent. Orchestration means low-risk systems can be patched automatically without additional involvement.

Next, the teams patching vulnerabilities need to understand the impact to their systems. That means knowing exactly what files will be affected by applying patches. The easiest way to accomplish this is to link the OEM release notes for each patch with the associated vulnerability so responders have a single source of information.

Another critical data point is the business context and impact for each vulnerability and how response activities will affect the business. Going beyond the IP address to know which items are vulnerable and what systems are dependent upon each vulnerable item gives responders the data they need to determine which risks are acceptable. Correlating business context with vulnerabilities also means you can create dashboards to instantly view your current exposure across the entire organization.

Business context can also be used to prioritize response efforts, allowing responders to focus on threats to critical systems first. Patches for less-critical systems can be postponed or automated as needed. This context also means you can see other systems that have the same vulnerabilities to identify potential targets for attackers.

The right solution also takes the guess work out of SLAs, using pre-defined processes for remediation. With workflows and routing, it's easy to track progress and see who's responsible for each task. Actions are documented so that processes can be improved in the future as well as providing historical audit data.

With the right solution in place, the vulnerability quarterback becomes part of the team, with everyone working from the same playbook. Using a single source of data combined with business context and prioritization means vulnerabilities can be patched efficiently and consistently. And when the next newsworthy vulnerability strikes, you can confidently know what systems are at risk without the chaos of phone calls, emails, and late-night text messages.