Why you shouldn’t deploy the Azure Monitor Agent on Client Devices, and the Cost of Closing Detection Gaps
In one of my blogs a very long time ago, I wrote about some discrepancies I found in the Defender file creation events in Microsoft Sentinel. At the time, I wanted to augment these file events with something more reliable like Sysmon, that would allow me to ingest high-fidelity events without any kind of filtering or event dropping by Microsoft. I turned to Olaf Hartong’s template for augmenting MDE events, which provided an opportunity to see what the experience was like for deploying the Azure Monitor Agent (AMA) to regular Windows clients. This proved to be an interesting but ultimately bad time for a number of reasons, and also ended up leading me down a somewhat philosophical path about the pros and cons of ingesting and paying for tons of data. This post is mainly about Microsoft Sentinel, but if you have a cloud SIEM that costs by the gigabyte, you might be interested.
TL;DR: Use a live response script to grab Windows events off the disk when you need them, or a tool like Velociraptor to interrogate them remotely on-demand. Don’t try ingesting this many events at scale in Sentinel unless you have infinite money (It’s probably still not worth it).
TL;DR;er: Would I recommend doing this? No, It’s a massive headache. Is it useful? To someone, maybe. Is it required? Almost certainly not.
Goodbye Log Analytics Agent!
With the Log Analytics Agent being phased out this year, Microsoft wants all its customers to switch from LA/MMA/OMS to AMA (Which hasn’t confused any customer at any point, ever, I’m sure). This comes with one massive caveat; AMA is currently only supported on Azure or Azure Arc managed resources. This leaves a conspicuous gap for resources that either aren’t managed in Azure, or are regular client devices. However, Microsoft do have a solution for this admittedly niche use-case.
The Process
What we need to do is this: we’ll need to create a new Data Collection Rule (DCR) and configure it to pick up our Sysmon events like you might a regular Azure server. Typically this involves creating the rule in the DCR wizard, assigning it to a resource, telling it what to pick, and away you go. This time though, we don’t have a resource to point it to. In fact, our AAD joined (they must be hybrid or AAD joined for this to work by the way) devices are completely absent in the wizard.
That is because we need to create a ‘Monitored Object’ as a stand-in for all client devices. This monitored object can ONLY be created at a tenant-level scope, and thus requires tenant-admin level permissions to create. Already, our first snag for major production environments — change control.
Once we’ve successfully got that past the change board, we’ll then need to associate our data collection rule against that monitored object in order for it to reflect back to the client device. These devices also require you to actually push AMA to them via some mechanism.
Creating the Monitored Object
As mentioned above, you do in fact need tenant-admin level permissions to create the monitored object. Once you’ve got these permissions, you can decide whether you want to use a PowerShell script or an API request to create the object as outlined in this section of the Docs.
Caveats
This is a niche use-case for many environments already, and Microsoft’s lack of features supporting it is reflective of that nicheness. You cannot:
- Change configuration options like Proxy configurations without a full reinstall of the agent
- Update the agent without a full reinstall
- Deploy to a non-joined device. No on-prem AD here!
This ultimately makes the process less than desirable from an IT SecOps point of view. You have a fleet of clients running an agent that cannot be managed directly, nor an agent that you can patch easily. You’re also sacrificing granularity as to what DCRs are applied to which device — it’s either all or none of them, thanks to the monitored object being applied at the tenant scope.
An Argument against AMA + Sysmon for Endpoints
That leads me to the hot-take section of the post. Does all of this effort pay off? Is it actually worth your teams time? I would say no, but bear with me.
Presuming you are a primarily Microsoft consumer running Sentinel and other Microsoft security products like Microsoft Defender, you would already have the option of ingesting all of the Advanced Hunting telemetry directly into Sentinel, and likely already are doing so. These events, [while flawed](Microsoft Defender for Endpoint Internals 0x03 — MDE telemetry unreliability and log augmentation | by Olaf Hartong | FalconForce | Medium), are often enough to cover most custom detections and threat hunting requirements on their own while also being fairly optimised from an ingestion and retention POV, even at massive scale. Sysmon on the other hand, without an extremely tuned config (more tuned than even Olaf’s MDE augment configuration), has a very good chance of nuking your costs for not that much additional benefit.
I’d also argue that collecting Windows Events (the other use-case for Azure Monitor Agent on clients) is not worth it either, for similar ingestion and cost-related reasons. If your organisation has the headroom for more ingestion, consider adding in network logging and ensuring you’re capturing events from as many kinds as devices as possible.
An Argument for AMA + Sysmon for Critical Endpoints
Not all devices are created equal of course. You might have a tier of servers that are critical, and thus require that extra level of detection capability. You might be doing some detection layering and desire those more granular events because the servers Sysmon is installed on are highly desirable targets. But in that case, you already have the option to install AMA on them through Arc or Azure directly! You can also manage these machines much more granularly without worrying about creating a tenant-wide monitored object.
An Argument for Neither
MDE filters out and drops events from the SIEM. That much we know is true. You will never get every instance of every event across every category of log in Sentinel or the Advanced Hunting pane — the bandwidth costs for Microsoft are just too large. So how do you address the gap?
Personally, I advocate for an in-between measure of installing Sysmon, deploying an MDE augment configuration or a fully featured one depending on the criticality of the endpoint, and configuring policies to retain that log for as long as possible locally on each machine. This means that you can build proactive detections using MDE’s dataset, and should the time come for a deeper investigation, you can pull those events off the box when you need them. This can be done forensically, through defender’s live response feature, or third party tooling like Velociraptor. While I’ll never say no to more data, we want good data. By ingesting millions of Sysmon events from a fleet of endpoints, you’re using up your ‘new log source’ budget for data that just isn’t going to be all that useful all the time.
While I was writing this, Microsoft had an outage that resulted in the raw telemetry not being available in Sentinel or Advanced Hunting. This is obviously a big issue, and a trade-off you inadvertently make when relying on any kind of SaaS. If you are particularly risk averse, that might be reason enough to roll out AMA and collect Sysmon events!
Closing Thoughts
To circle it all back around to AMA again, I hope this has given you some insights into not only the “how and why” of deploying AMA to all your clients, but also the “if”. Detection layering, not just the use-cases themselves but also the underlying data sources is an important point of discussion and isn’t often talked about enough when dealing with massive environments. I would encourage you to have a read of the EDR Telemetry project, and seeing for yourself if the telemetry you’re getting is enough for your detection needs.
If you’re in that small minority of orgs who simply must have reliable event ingestion at any cost, AMA is there for you. Kinda. See above 😉.
