2022-Jun Duplicate Agent Incident (Resolved)

Team: Huntress EDR
Product: Huntress Agent
Environment: Huntress Agent version 0.13.12
Summary: Agent version 0.13.10 contained an issue resulting in duplicate agents.

There is no action needed from you at this time. We are actively working on this issue.

Please follow this page to get updates on our progress.

Update 2022-06-13 09:00 ET

Duplicates in Portal

We've eliminated 99%+ of duplicates. We are continuing to monitor the situation and are investigating remaining edge cases to determine if we need to make any additional changes to our algorithms. 

Special Note for Resellers and Accounts purchasing under a Reseller:

Resellers (and the charges for their accounts/contracts) get billed for overages, so accounts under resellers that may have overages were not overbilled. 

Update 2022-06-10 09:00 ET

Duplicates in Portal

We are making progress with the rollout of our de-duplication script. We are more than halfway through removing the duplicates. If you are still seeing duplicates, don't worry we will get to you. Thank you for your patience as we work through this issue.

Update 2022-06-09 07:00 ET

Agent

Huntress has squashed the bug that generates the duplicates.

Duplicates in Portal

We have created and finished testing an automated script that works to fix/remove the duplicated agents for a given organization. 

Billing

We have a programmatic way we’re working on to better output the number of over-billed agents per account—we believe we can figure out a programmatic way to create [invoice] credits for the affected accounts. Automating the credits on Partner bills would hopefully reduce the burden on Finance and result in our Partners getting credits faster. 

What's next?

The next steps are to run this automated process that will clean up the duplicated agents. We will continue to monitor the progress and post updates to this page as needed.

2022-06-07

TL;DR:

Huntress is aware of an issue where our agent/portal generates duplicate agents for some environments. 

Duplicate Agent Explanation

Huntress is aware of and working to address an issue with a recent agent update that resulted in the creation of duplicate agents in the Huntress portal. This issue has impacted 4% of total deployed agents. This issue will be resolved without any action needed from our partners. Affected partners that were over-billed due to this issue will receive a refund for their overpayment. Partners billed after issue resolution will not be affected.

This issue relates to Huntress’ inclusion of a device’s last boot time in determining a unique ID for each agent. With agent v.0.13.10, we noticed inaccuracies in the Windows system call the Huntress agent utilizes to determine the system boot time. In agent v.0.13.12, Huntress leveraged a more accurate Windows system call to determine device boot time. 

Unfortunately, Huntress did not anticipate high variance times between the two different system calls. In testing, we determined that a variance time of approximately 5 seconds would sufficiently cover all currently deployed agents. For some of our agents, the variance exceeded 5 seconds. This resulted in duplicate agents within our portal.

Huntress is actively working to rectify this issue within the portal and will notify partners when the fix has been rolled out. Huntress services and Security Analysis were not affected by this issue.

What’s happening?

Huntress is aware of and working to address an issue with a recent agent update that resulted in the creation of duplicate agents in the Huntress portal. This issue has impacted 4% of total deployed agents. 

This issue relates to Huntress’ inclusion of a device’s last boot time in determining a unique ID for each agent. With agent v.0.13.10, we noticed inaccuracies in the Windows system Call the Huntress agent utilizes to determine the system boot time. In agent v.0.13.12, Huntress leveraged a more accurate Windows system call to determine device boot time. 

How does this affect me?

Huntress services and Security Analysis were not affected by this issue. Affected partners that were over-billed due to this issue will receive a refund for their overpayment. Partners billed after issue resolution will not be affected.

What do I need to do?

This issue will be resolved without any action needed from our partners. 

What is Huntress doing to fix this?

Huntress is actively working to rectify this issue within the portal and will notify partners when the fix has been rolled out. We will be scrubbing duplicate agents and issuing refunds/credits where applicable.