A month ago, we talked about how visibility can make us more frustrating victims to our adversaries. It makes sense – easy marks are those who don’t see that they are victims in the first place! Take victims of physical (traditional) crime. Burglers love a target who isn’t using alarm systems, cameras, or even their own eyes and ears to actively detect incursions. But having eyes and ears isn’t what makes you formidable. It is that you have those sensory inputs AND you know how to interpret what they are saying and how they respond. Do you know how to discern bad behavior from the norm and know how to tell between friend and foe? And do you know what the right response is based on that proper interpretation? We’re going to tackle the first question here today as we discuss how killer baselines improve security outcomes.

Home alone?

Anyone with an off-the-shelf home security camera solution knows that they are false positive machines in the early stages. Assuming you have the patience to tune them, you configured zones to tell the ‘artificial intelligence’ what areas to ignore. You might even be lucky enough to have a system that can discern a human from a cat|bird|leaf. If you’re lucky, they’ll tune those alerts down even further. While this seems tedious, it is easier in a physical environment because falling leaves pose no major threat. A lurking human wearing a mask at 1AM probably needs your attention. I don’t know your threat picture, but I am pretty confident that we all would have some questions there.

Your alarm systems and even neighbors are also ‘tuned’ to expect certain behaviors. Assuming each is accustomed to your schedule or habits, they’ll permit what is normal, predictable, and acceptable behavior. But if you are away for the weekend and someone comes snooping around, good versions of either should be expected to raise the alarm, or at the very least increase scrutiny. This is good stuff! We like that. We may even put up with occasional false alarms (false positives) in exchange for vigilance.

It should be no surprise that visual evidence of alarm systems and conscientious neighbors are not just good at identifying malicious behavior, but in deterring it. Neighborhood Watch programs demonstrate a significant impact, with estimates of 16% reduction in crime vs. non-Watch areas. Similar effects arise when an adversary sees clear signs of an alarm system. The point is that your security ‘stack’ has situational awareness that is dialed in – the baseline is well understood. Both you and potential adversaries should see the value in that an plan accordingly.

Getting to total situational awareness

Like our homes, our information systems offer many areas for visibility, detection, and control. Unlike our homes or brick-and-mortar businesses, however, the diversity of telemetry and volume of events will crush an unassisted human analyst. (I can only imagine the names I would be called if I put my neighbors in charge of a security solution). So how do we manage that workload and make sense of all of that noise? We may be tempted to leverage a SIEM, deploy an XDR, or hire an outside firm, but these all treat the symptoms and alone are not enough. David Bianco helped us out with a fantastic maturity model that offers a solid conversation starter. Some of us may want to chase Archer’s dream of Total Situational Awareness, but YMMV.

David Bianco’s Threat Hunting Maturity Model is a fine place to start your visibility and detection discussion (from his whitepaper here)

Setting realistic goals

What is your end goal? Are you hoping to proactively threat hunt? Automate your responses? Or do you have compliance requirements for collection that may be leveraged in forensics? You should also consider your team in the broadest sense of the term. Will an MDR or MSSP be helping out? Do you have an IR firm on retainer? In any case, what will those stakeholders need to get their jobs done and help protect you?

The key here is to deploy the visibility you and your stakeholders need to monitor effectively, and no more. We might all be tempted to tackle 100% coverage of an ATT&CK-based threat picture, but if getting there requires your organization to consume so many logs that your analysts are unable to analyze, then you have just reduced your security posture, not improved it. The goal should be to give your team enough information to understand what is normal, such that they can then ‘identify the strange’. How do we do that?

What do we need?

What makes sense will vary from environment to environment, but you and your team typically need something from each of the following categories:

  • Endpoint telemetry (AV, EDR, Windows Events, Syslog/Rsyslog/etc.)
  • Network telemetry (firewall events, flow data, connection/transaction logs, etc.)
  • Administrative telemetry (domain, authentication, services, etc.)
  • Infrastructure telemetry (network devices, servers, power, backup)

All of the above helps paint a picture, but that picture will be much richer when build with some additional building blocks:

  • A solid understanding of your inventories (hardware, OS, applications, software supply chain)
  • An accurate assessment of your vulnerabilities and exposures
  • A system or systems that can support an analyst in making sense of the above sources
  • Time (sorry folks, it is unavoidable! Time to do the job, to train, and to deploy properly)

If you are uncertain as to what telemetry sources make sense, think about the systems you will be feeding that telemetry into and work backwards from there. folks like Florian Roth have blogged extensively on what sorts of events offer the greatest return on investment, and your security solutions (SIEM, XDR, UEBA, MDR) will all have capabilities that lean more toward a subset of the above sources for higher efficacy. His Log Sources table below has been a go-to resource for me in working with both customers and product teams.

Florian Roth’s awesome take on relative values of telemetry sources. Not complete, but its a great start! (from his GitHub repo here)

Is it enough?

Now if you are serving web applications, add some access logs in there. Using a proxy? Ship those logs too! The point is that you must know yourself and from there determine what sorts of telemetry you must have to detect variations in your environment’s health. You don’t need ALL of the above, but should instead focus on the easiest-to-digest sources that your staff can handle that will also best help flesh out our baselines and detection tools.

What if you find you can’t consume enough to truly know your environment and establish that baseline situational awareness? This is where the tough decisions happen. If you are securing a money-making enterprise or otherwise supporting a business, effective security must be table stakes. The years of ignoring this responsibility are over – we are all assured of being hacked or breached. How we respond and how quickly we return to operations hinges on how seriously we take gaining effective situational awareness. Time and time again, organizations that short their security practices are not only more rapidly breached, but find they are also unable to respond and recover. A cyber event becomes an extinction-level crisis.

The flip side? Collecting too much is destructive in another sense. I have seen many organizations experience paralysis by analysis that comes from drowning in data. Not only does this prevent the team from seeing the important events clearly, but the organization wastes considerable time, money, and resources supporting a bloated collection approach that is them blamed for the lack of efficacy. A right-sized telemetry approach truly helps wrangle those SIEM costs and ensures that your entire stack is more effective.

It’s all about the base(line)

Once we’ve gotten the telemetry we need, our chosen monitoring/collection tools will dictate how the baseline is built. The most basic approach is to record relevant telemetry across a specific time slot, offer some tweaks or tuning to recommended rules, and to then leverage that baseline in identifying malicious behavior. These baselines are in place to allow tools to provide custom detection . These baselines improve security outcomes by incorporating your local environment’s context. This brings a new dimension to the signature or Indicator of Compromise (IoC) based detection capabilities inherent in each tool.

There are some factors that may hurt our results, however:

  • The dataset may be too small to build an effective baseline (#1 issue for home labs and small businesses)
  • The baseline may be built on an already compromised environment (thus baking in malicious behavior as “normal”
  • The baseline may be too rigidly defined – retail businesses , for instance experience massive swings in activity throughout the calendar year, and US health insurers during Open Enrollment.

Improving on imperfect baselines

The issues above are inherent to any baseline, but we are seeing vast improvement with the inclusion of more sophisticated machine learning and AI deployments. Even if we can get past that issue, we have to wrestle with the inconsistencies across our security stack. Traditional building blocks like EDR and NDR tools will typically have different time epochs, for instance, that they use to observe behaviors before gaining the confidence needed to flag potentially malicious behavior. On the other hand, firewalls and SIEM’s might offer more customization, and it may be a variable considered in the building of the analytic or signature. If we ignore these nuances, we doom our program to continually let us down. Savvy organizations will turn these overlapping rhythms into advantages that allow each tool to help cover blind-spots in others.

Wrapping up

Hopefully this post has helped you see how baselines improve security outcomes for you and your environment. We should learn enough about what telemetry best builds that situational awareness for our tools and meets the needs of all teammates (in-house or 3rd party) and do our best to feed it exactly and only what it needs to be most effective.