On July 19, 2024, a CrowdStrike update caused unexpected issues for Microsoft Windows users due to a stop code related to the csagent.sys file. Key symptoms included hosts experiencing a bug check or blue screen error related to the Falcon Sensor.
Details of the Impact
The problematic version of the channel file, identified as "C-00000291*.sys" with a timestamp of 0409 UTC, caused these issues. However, certain environments remained unaffected:
- Windows hosts brought online after 0527 UTC.
- Hosts running Windows 7/2008 R2.
- Channel file "C-00000291*.sys" with a timestamp of 0527 UTC or later.
- Mac- or Linux-based hosts were not impacted.
Resolution Steps
For Individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
1. Boot Windows into Safe Mode or the Windows Recovery Environment- NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys” and delete it.
4. Boot the host normally.
- Note: BitLocker-encrypted hosts may require a recovery key.
For Public Cloud or Virtual Hosts:
To address the issue, organizations can follow one of the two remediation options:
Option 1: Manual Fix
1. Detach the operating system disk volume from the impacted virtual server.
2. Create a snapshot or backup of the disk volume as a precaution.
3. Attach the volume to a new virtual server.
4. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
5. Locate and delete the matching “C-00000291*.sys” file.
6. Reattach the fixed volume to the impacted virtual server.
Option 2: Snapshot Rollback
- Roll back to a snapshot taken before 0409 UTC
For organizations with BitLocker, the security key will be required during the remediation process. Additionally, an automated CrowdStrike workaround in Safe Mode using Group Policy Object (GPO) is available.
Impact on C1 Customers
Given C1’s close ties with our technology partners, we receive early notification regarding potential issues and remediation efforts. This enables C1 OnGuard Managed Services to maintain active monitoring, provide early support and minimize potential downtime. Customer impact for this recent outage was minimal. Intermittent issues were easily addressed due to our tailored Managed Services design deployed for each customer.
Stay tuned for further learnings based on this and other recent incidents.
