July '22 Unresponsive Agent Issue

Team: Huntress EDR
Product: Huntress Agent
Environment: Huntress Agent version 0.13.12
Summary: Agent version 0.13.12 contained an issue causing unresponsive agents and updaters. This was resolved with agent version 0.13.16.

In this article

1What happened?

2What does it mean if some of my agents are unresponsive or deleted?

3Why did this happen?

4Action Needed from Partners

5What is the timeline for this incident and our response?

6FAQ

What happened?

• An agent deploy (version 0.13.12) on June 3rd, 2022 contained an issue that may have caused the agent and updater on some systems not to start or run reliably after a reboot, resulting in them becoming unresponsive (they weren’t sending data to the cloud for ThreatOps analysis).
• For Partners utilizing the Unresponsive Agent Settings within the Portal, agents that have been in an unresponsive state for longer than the configured time will have been removed.
  Partners can configure "# days until unresponsive" AND "# days before unresponsive action" within the Huntress portal

What does it mean if some of my agents are unresponsive or deleted?

• For the agents that have been unresponsive or deleted, the Huntress portal has no visibility into device activity and is unable to detect potential compromise. Preventative functionality (Windows Defender or existing AV product) still behaves normally.

Why did this happen?

• There was an issue introduced in one of our agent updates
  • This impacted how the agent starts, sometimes causing it to try to use a specific Windows service before that service was ready.
  • When this happened, the agent and updater would both fail to start, leading to them being unresponsive.
  • Agents that are unresponsive may be deleted after a certain period of time (Starting at 21 days - configurable in your settings).

Has this been fixed?

• Yes! Agent version 0.13.16 and above fixes the issue.
  • Huntress has taken action to update all agents that are actively communicating with the portal.
  • To check to see if there are any agents still at risk; visit the “Agents” page and apply the “outdated” filter. If any agents appear in this list they will need to be updated to the current agent version. While agents prior to 0.13.12 are not impacted we highly recommend ensuring these agents are also on the current version.
• The agent now explicitly waits for the necessary windows service to be available prior to attempting to use it, eliminating the possibility of race condition that resulted in a failed start

Action Needed from Partners:

• We need partners and customers to restart unresponsive agents and updaters manually or through their RMM. For Partners using their RMM Tool, please download the latest version of our RMM Deployment Script specific to your RMM Tool from here as we have updated them to include the fix for this issue, previous versions will not restart unresponsive agents or updaters. Note that in some cases, the new Deployment script may trigger the agent to uninstall once run, if this should happen please re-run the Deployment Script.

What is the timeline for this incident and our response?

• On 06/20/2022 we started receiving reports of what we believed to be a localized issue presenting in a small number of accounts. A manual fix to restart the services was immediately created and shared with partners that had reported the issue.
• On 07/11/2022 engineering released agent version 0.13.16 which resolved the issue without the need for a manual fix.
• Internal reporting on 07/13/2022 indicated an uptick in agents becoming unresponsive. A task team was pulled together to identify the root cause and resolve as quickly as possible.
• We sent a communication on 07/15/2022 asking for Partners to update their agents due to what we thought at the time was a bug in a Windows update.
• We continued to investigate the issue to further confirm our hypothesis and on 07/21/2022, we identified the root cause was introduced internally in the 0.13.12 release (released 06/03/2022).
• We are updating our Partners now with the new information as we have been able to identify which accounts were impacted and an action that needs to be taken.