Hypervisor-based malware is a particularly tricky situation to handle because a vast majority of the malicious activity that would traditionally be picked up by an EDR solution happens in two places: 1) the hypervisor’s configuration application or website (e.g. vSphere); and 2) directly on the hypervisor’s operating system.These 2 areas fall outside of an EDR’s capabilities for a couple different reasons:
- First, hypervisor configuration applications do not offer any data sources that could be collected and monitored by an EDR solution. Typically these applications are graphical interfaces hosted in a web browser or an application installed on an endpoint without any of the programatic access required to monitor what the application is doing.
- Second, it is generally not recommended or supported by the hypervisor vendors to run 3rd party agents on the operating system. For example, Broadcom’s (VMware) support article specifically states “…it also makes ESXi unable to run “off-the-shelf” software, including security tools…”
Because of these 2 limitations, EDR solutions focus on finding malicious actors on traditional endpoints before they are able to get access to the hypervisor. While we pride ourselves on our ability to quickly find and stop malicious actors, the reality is that some behaviors related to hypervisor-based ransomware are so close to normal administrative activity that it’s difficult to have a firm malicious conviction.Note: in the following section we could add specifics about individual cases that showcase where we did (or did not) pick up on malicious activity. In the case referenced above, we could highlight that a known-good ScreenConnect instance was compromised which basically let them walk in the front door and probably straight to the hypervisor apps.Typically the tradecraft associated with these attacks looks something like this:
- The malicious actor gets a foothold in the environment
- This can happen in a number of ways including malware installed on a system, brute forcing publicly accessible RDP connections, compromising a VPN account, etc.
- They enumerate the environment to figure out if there's a hypervisor present
- This is generally a fast and low noise activity
- They figure out a way to get credentials to the hypervisor. Various ways of doing this. In some cases they already have credentials.
- Passwords compromised via infostealer
- Brute force
- Exploiting AD-joined ESXi servers via ESX Admins group
- Looking through internal documents or portals for plaintext passwords
- Once they have credentials to the hypervisor, EDR products lose their visibility
- Next they will log into the configuration application, do some light enumeration, and enable SSH
- Once enabled, they will log into the hypervisor’s operating system directly
- They either detonate their ransomware payload or they'll use LOLBins (e.g. openssl) to do the encryption that way
Hypervisor-based malware attacks are generally considered rare; however, we are investigating additional ways to stay ahead of this tradecraft. One promising avenue is feeding the hypervisor’s logs into our Managed SIEM product and using them as a source of telemetry to look for brute-force authentication attempts, SSH being enabled on the hypervisor, and possible enumeration techniques.Additionally, we also recommend following a defense in-depth strategy. For hypervisors specifically we recommend the following:
- Ensure that the hypervisor's management interfaces are only accessible to a minimal number of internal hosts and verify that they are not publicly accessible,
- Disable any unnecessary hypervisor management interfaces (such as SSH),
- Enforce strong passwords for hypervisor management credentials,
- Enforce MFA for hypervisor management credentials,
- Regularly review hypervisor group memberships and limit access to only absolutely necessary administrators