Summary

  • A senior executive from cyber-security firm CrowdStrike has apologised to US lawmakers at a hearing into the global IT meltdown in July

  • Adam Meyers, a senior vice president, says "we are deeply sorry", and that his firm is "determined to prevent it from happening again"

  • Millions of people were impacted by the massive outage after the company released a faulty software update which froze Windows-operated computers

  • During the outage, GPs couldn't treat patients, planes couldn't get off the ground, and small businesses lost sales

Media caption,

CrowdStrike executive apologises for global IT outage

  1. Exec explains how CrowdStrike will try to prevent another global IT outagepublished at 21:50 British Summer Time 24 September

    Zoe Kleinman
    Technology editor

    This was definitely one of the calmer and more measured tech committee hearings I have sat through.

    CrowdStrike described the July 19 global outage as the result of a “perfect storm”.

    The fundamental measure it has now put in place to prevent a repeat performance is a pretty simple one. It is doing a lot more internal testing of its updates (I learned a new term for this: “dog fooding”) and also enabling customers to decide when to accept them, so that they are no longer released simultaneously.

    With 10-12 updates pushed out every day, this is almost certainly going to slow down CrowdStrike’s operations.

    Adam Meyers was articulate and measured in his responses. I don’t think I saw him take a sip of water during his 1.5 hours in the ring. He apologised more than once for the firm’s failings, and stressed that it had not been a cyber attack, although he added that threat actors were definitely trying to leverage it.

    The assembled lawmakers also shared some of their own vulnerabilities.

    Congressman Carlos Gimenez said he was afraid of AI and asked about the threat of it writing malicious code. CrowdStrike’s Adam Meyers said he thought the tech was “not there yet" but added that every day it "gets better".

    Congressman Gonzales said he feared that Congress itself didn’t understand the wider issue, and asked how the US government could improve its own response during a similar situation.

    Adam Meyers said during the CrowdStrike outage, it was the firm's job to inform government, during a cyber incident it would be to support government. “This is a team sport and we are all on the same team” he said.

    You can read more about the impact of CrowdStrike's outage here.

    We're now finishing out live coverage. Thanks for joining us.

  2. A ruined surprise trip to Disneylandpublished at 21:31 British Summer Time 24 September

    Zoe Kleinman
    Technology editor

    Yesterday I spent some time with Jo, a lady from Fife, on the east coast of Scotland, who works part-time as a carer, and has two small children.

    She saved £300 ($402) a month for six months to take them to Disneyland as a surprise. They were due to fly on her birthday, then CrowdStrike happened.

    The flights were grounded, and after 14 hours in the airport, their re-booked flight was also cancelled. Her travel costs were refunded but she lost money on the theme park tickets bought from a third-party website.

    Jo showed me this picture of her kids, beaming in Mickey Mouse ear headbands and Marvel outfits on the way to the airport on 19 JulyImage source, Supplied

    Jo had never heard of CrowdStrike, or “Blue Screens of Death”, or cyber security update glitches.

    When I explained some of it to her, she couldn’t comprehend how something so far removed from her life had ruined her family surprise.

    Today, CrowdStrike exec Adam Meyers met with a US Congress subcommittee to explain what happened, and to apologise to those people impacted by the outage.

  3. CrowdStrike hearing endspublished at 21:06 British Summer Time 24 September

    That's a wrap from Congress.

    We had about 1.5 hours of testimony from CrowdStrike executive Adam Meyers, with the main line being his apology for the global IT outage.

    Stick with us, we will bring you some more updates shortly.

  4. 'Trying to move a chess piece where there's no square'published at 20:44 British Summer Time 24 September

    When talking about what caused the outage, Meyers has used an interesting analogy to explain why CrowdStrike’s testing mechanism failed to detect a glitch in its content update file, when pressed to by Representative Morgan Luttrell.

    Given the challenge of summing it up in just over a minute, he likened the issue affecting its Falcon sensor - used to detect threats to systems - to a game of chess.

    “If you think about a chessboard trying to move a chess piece to someplace where there's no square, that's effectively what happened inside the sensor,” he said.

    This meant that when its system tried to carry out the instructions or rule contained within the file, it wasn’t able to carry it out and this “triggered the issue within the sensor” he added.

    In its root cause analysis, external designed to provide an in-depth post mortem of what went wrong, published in August, the company said the issue affecting the file in question "is now incapable of recurring".

  5. CrowdStrike - the impact on healthcare, travellers and small businessespublished at 20:19 British Summer Time 24 September

    Graham Fraser
    Technology reporter

    Delta Airlines staff search through passenger luggage at Los Angeles International Airport on 24 JulyImage source, Getty Images

    Lawmakers have been pointing out some of the ways the CrowdStrike outage affected normal operations - for governments, businesses and regular people.

    Doctors couldn't treat patients, people were stranded as planes couldn't get off the ground, and small businesses lost much-needed sales.

    These were just some of the impacts of the CrowdStrike outage when it hit the world on 19 July.

    Dr David Wrigley, a GP in England for the past 22 years, told me how referrals for possible cancers were delayed by days as the online systems doctors like him rely on went dark.

    "It was a very difficult period of time with very little help and support," he said.

    I also heard from others in healthcare, UK holidaymakers who were left stranded for days, and business who lost lots of money thanks to the CrowdStrike outage.

    "It showed how many systems worldwide had put their eggs in one basket," one London businesswoman told me.

  6. 'AI was not responsible for making any decision', CrowdStrike exec sayspublished at 20:10 British Summer Time 24 September

    Zoe Kleinman
    Technology editor

    When tech leaders get summoned before lawmakers, it can get pretty brutal.

    The committee members get five minutes each to ask questions, and generally, they tend to go for the jugular – just ask Mark Zuckerberg.

    But so far, especially considering the sheer scale of the global chaos CrowdStrike accidentally caused back in July, it has been fairly measured.

    The committee says it appreciates the work of CrowdStrike. Congressman Eric Swalwell said they have not gathered to “malign” the firm and chairman Mark Green says Adam Meyers displays “a degree of humility that is impressive”.

    If he was hoping for a gotcha moment when he asked whether the decision to push out the faulty update had been made by an AI tool, he didn’t get one.

    “AI was not responsible for making any decision” replies Meyers, adding that CrowdStrike released 10-12 updates every day.

  7. What is ‘The Kernel’ and why is it important?published at 20:06 British Summer Time 24 September

    Joe Tidy
    Cyber correspondent

    Among discussions about how to prevent another disaster, one term keeps coming back: The Kernel.

    And US lawmakers have just asked Adam Meyers about how the kernel was related with this year's outage.

    The Kernel is the name for the core of an operating system. If a computer is an oak tree, its kernel is the largest, deepest and strongest roots of that tree.

    Security products like CrowdStrike’s Falcon software have privileged access to those deepest parts of the system to allow it to detect small changes in activity which might be cyber attacks.

    CrowdStrike and other security providers cherish kernel access as it helps them make better products.

    But that privilege can also have catastrophic consequences when something goes wrong. If an app with normal surface level access on your computer or phone crashes, then you can usually close it down and start again.

    If an app with kernel access crashes – the whole machine falls over.

    Since the disaster Microsoft and others have been weighing up the need for this privileged access versus the risk.

    There is a tussle taking place right now in the industry and CrowdStrike is fighting hard to maintain its access in fear of being “kicked out of the kernel” which could upend the way they build products.

  8. 'We are deeply sorry' – CrowdStrike execpublished at 20:01 British Summer Time 24 September

    Media caption,

    CrowdStrike apologises for global IT outage that affected 8.5m computers

    In his opening testimony, CrowdStrike executive Adam Meyers apologised for the impact of July’s mass outage.

    "On behalf of everyone at CrowdStrike, I want to apologise," Meyers tells representatives – reading from his written testimony published ahead of the hearing, external.

    "We are deeply sorry this happened and are determined to prevent it from happening again."

    He says the company would be making sure it acted on and shared “lessons learned” from the incident to ensure it wouldn’t happen again.

    This also included improving its testing and checks on updates, and changing the way it issued them to prevent similar issues.

    He’s now being questioned by representatives. Committee chair Mark E. Green has made a point of saying they would’ve liked to see the company's CEO George Kurtz at the hearing instead.

  9. CrowdStrike exec apologises for global IT outagepublished at 19:54 British Summer Time 24 September

    Adam MeyersImage source, US Pool

    In his opening statement, senior executive Adam Meyers has told US lawmakers "we are deeply sorry this happened".

    He also said the firm was "determined to prevent it from happening again".

    He read from his written statement, which was shared before the hearing.

    We'll bring you more of his remarks shortly.

  10. ‘A catastrophe that we would expect to see in a movie’published at 19:42 British Summer Time 24 September

    Mark E. Green, chairman of the House Homeland Security Committee, says July’s mass global IT outage “is a catastrophe that we would expect to see in a movie” in his opening statement.

    The representative for Tennessee went on to say that the widespread impact of CrowdStrike’s faulty content update was the sort of thing “we would expect to be carefully executed by malicious and sophisticated nation-state actors”.

    “To add insult to injury, the largest IT outage in history was due to a mistake,” he says.

    “Mistakes can happen. However, we cannot allow a mistake of this magnitude to happen again”.

    Green says US national security hinges on its partnership between public and private companies to safeguard networks in the face of threats.

    “Ensuring our partnership is strong is important because our adversaries always watch how we respond to major incidents like the July 19th outage.

    "You can bet that they’re watching us right now.”

  11. 'The scale of the outage was alarming'published at 19:36 British Summer Time 24 September

    The hearing at the US Congress begins with representative Andrew Garbarino reading out an opening statement outlining what unfolded on 19 July.

    He goes on to say that the committee is gathered today to find answers about what went wrong and how such outages can be avoided in the future.

    The scale of the outage was alarming, he says, before going on to warn of how malicious actors could take advantage of such instances in the future.

  12. Hearing beginspublished at 19:33 British Summer Time 24 September

    Things are getting underway at Congress, with subcommittee chair Andrew Garbarino introducing the witness.

    You can watch the hearing live at the top of this page.

  13. CrowdStrike exec arrives at hearingpublished at 19:23 British Summer Time 24 September

    Lily Jamali
    North America Technology Correspondent, reporting from the hearing

    Adam Meyers (centre) arrives at the hearingImage source, Lily Jamali / BBC
    Image caption,

    Adam Meyers (centre) being greeted by Subcommittee Chair Andrew Garbarino

    Adam Meyers has just walked in. We ask him twice why he is here instead of George Kurtz, CrowdStrike's CEO.

    He did not take questions and just looked straight ahead.

    A press handler said Meyers would not be answering questions from the media today.

  14. 'We are deeply sorry,' senior executive sayspublished at 19:16 British Summer Time 24 September

    Brandon Drenon
    Reporting from Washington DC

    We are still waiting for the hearing to start. It was scheduled to begin 15 minutes ago, but a political staffer has told us there's been a slight delay.

    When it does begin, Adam Meyers, a senior executive at CrowdStrike, will be offering his apology for the company’s massive tech failure.

    "We are deeply sorry this happened and are determined to prevent it from happening again," he said in a written testimony, external released ahead of this hearing.

    "We have undertaken a full review of our systems and begun implementing plans to bolster our content update procedures so that we emerge from this experience as a stronger company.”

  15. The biggest IT outage in historypublished at 19:05 British Summer Time 24 September

    Joe Tidy
    Cyber correspondent

    The CrowdStrike outage was the biggest cyber event in history, 8.5m Windows computers crashed.

    Vital computers used by banks, airlines, shops, transport services were all disabled.

    The crash meant that IT experts had to get hands on with computers to get them back up and running, which added to the chaos of the situation.

    The general trend I observed was that the largest and best resourced companies tended to get back on their feet first – most within a day.

    But the smaller organisations which relied on outsourced IT teams or depleted staff struggled the most. It took a week for 97% of computers to be back up and running, according to CrowdStrike.

    The closest cyber event to this was the WannaCry cyber attack in 2017 that is estimated to have impacted about 300,000 computers in 150 countries.

    There was a similar costly and disruptive attack called NotPetya a month later.

    There was also a major six-hour outage in 2021 at Meta, which runs Instagram, Facebook and WhatsApp – but that was largely contained to the social media giant and some linked partners.

  16. Top exec to testifypublished at 18:58 British Summer Time 24 September

    We are expecting the hearing to start in the next few minutes. You will be able to watch it live at the top of this page, and we will bring you text updates and analysis throughout.

    So, who are we going to hear from?

    Adam Meyers - a senior executive at CrowdStrike - will testify to explain what happened and how the company is going to prevent another IT disaster.

    He is CrowdStrike’s senior vice president of counter adversary operations and oversees threat intelligence, according to the company’s website, external.

    This will not be Meyers first time before Congress. In 2016, he participated in a Congressional hearing on ransomware conducted by the Senate Judiciary Committee’s Subcommittee on Crime and Terrorism.

    George Kurtz, CrowdStrike’s CEO, will not testify, frustrating Congressional leaders who sent him a letter in July asking him to appear.

  17. What is CrowdStrike?published at 18:50 British Summer Time 24 September

    Liv McMahon
    Technology reporter

    CrowdStrike is a cybersecurity firm founded in 2011 with the aim of safeguarding the world’s biggest companies and hardware from cyber threats and vulnerabilities.

    It specialises in endpoint security protection and tries to prevent malicious software or files from hitting corporate networks from devices that connect to them, such as phones and laptops.

    It also aims to protect the data of companies which have shifted from guarding it under their own roof, or on their own servers, to so-called cloud providers.

    The Texas-based firm was co-founded by entrepreneurs George Kurtz, who remains chief executive, and Dmitri Alperovitch.

    In 2016 CrowdStrike was called in by the US Democratic National Committee, the strategy arm of the Democrat Party, to investigate a breach into its computer network.

  18. What is the Blue Screen of Death?published at 18:44 British Summer Time 24 September

    Liv McMahon
    Technology reporter

    The Mercedes Formula One team, which is partnered with CrowdStrike, faced error screens ahead of practice sessions at the Hungarian Grand Prix in July.Image source, Getty Images

    PC users will know there are a whole host of error messages that can cause panic and frustration.

    But those of us who have struggled with Windows’ so-called blue screen of death (BSoD) before will feel the pain of those who were met with it during July’s CrowdStrike outage.

    It became the face of the mass IT incident, with bricked computer displays around the world, launching a thousand memes online.

    The screen appears when there is a critical error affecting the operation of your PC – and can trigger an unending loop of trying and failing to restart and launch properly.

    A BSoD can be prompted by hardware or software issues, but it's not the easiest thing to resolve – and often demands physical fixes.

    Microsoft estimated that 8.5 million Windows devices were affected by CrowdStrike’s update.

  19. What happened in the CrowdStrike meltdown?published at 18:40 British Summer Time 24 September

    On 19 July, an outage caused by the technology security company CrowdStrike crippled roughly 8.5m computers globally.

    The incident occurred after CrowdStrike sent out a corrupted software update to its huge number of customers.

    Subsequently, Microsoft Windows systems failed almost instantly for millions of users as a result.

    The outage is estimated to have cost $5.4b (£4b) for US Fortune 500 companies alone.

  20. CrowdStrike executive to apologise for global IT outagepublished at 18:38 British Summer Time 24 September

    Brandon Livesay
    Reporting from New York

    One of the first signs something was drastically wrong was mass flight cancellations on airport departure boards around the world.

    As the cancellations spread, so did the realisation that this IT issue was impacting much more than just airlines.

    The CrowdStrike IT outage on 19 July crippled about 8.5 million computers which were using Microsoft systems. Doctors were unable to see patients, small businesses lost customers and many others were left in the lurch, thanks to a rogue software update by the US cybersecurity company.

    Today, Adam Meyers - a senior executive from CrowdStrike - will appear before a US Congress subcommittee.

    He will apologise for what happened, and give his explanation of how similar issues will be avoided in the future.

    We'll be bringing you updates from the hearing, which you can also live stream at the top of this page.

    Stick with us.