Full story: Amazon Web Services outage hits dozens of websites and apps
Dan Milmo
A major internet outage has hit dozens of websites and apps around the world, with users reporting troubles getting online after problems at Amazon’s cloud computing service.
The affected platforms include Snapchat, Roblox, Signal and Duolingo as well as a host of Amazon-owned operations including its main retail site and the Ring doorbell company.
In the UK, Lloyds bank was affected as well as its subsidiaries Halifax and Bank of Scotland, while there were also reports of problems accessing the HM Revenue and Customs website on Monday morning. Also in the UK, multiple Ring users took to social media to complaint their doorbells were not working.
In the UK alone reports of problems on individual apps ran into the tens of thousands for each platform. More here:
Key events
Amazon’s cloud services unit AWS is struggling to recover from a widespread outage that knocked out thousands of websites along with some of the world’s most popular apps – Snapchat and Reddit – and disrupted businesses globally, Reuters reports.
After more than six hours of disruptions, some applications were gradually coming back online as of 10:00 a.m. ET (3pm UK time). But AWS acknowledged it was still experiencing elevated errors.
In its latest status update, AWS says:
“We can confirm significant API errors and connectivity issues across multiple services … We are investigating,”
To aid the recovery, AWS said it was putting in place limits on the number of requests that can be made on its platform.
While some apps like Reddit and Roblox had largely stabilised, according to outage tracking website Downdetector, others, including Snapchat and Duolingo, were showing a resurgence in issues seen earlier in the day.
Indeed, this chart shows how reports of outages at Snapchat rose an hour ago:
Treasury Committee: Is government concerned about dependance on overseas IT?
UK politicians are demanding answers over the disruption at Amazon Web Services today that has grounded many website and apps today.
The House of Commons’ Treasury Committee has written to the Economic Secretary to the Treasury, Lucy Rigby, to ask why the government has not, yet, designated Amazon a “critical third party” to the UK’s financial services sector.
Committee chair Dame Meg Hillier points out that Amazon recently told the committee:
Financial services customers are choosing to use AWS to support and improve their security and resilience. AWS has a comprehensive approach to resilience that spans multiple layers of protection, ensuring businesses can reliably maintain operations.
That claim rings a little hollow, now that Lloyds Bank customers were unable to access their accounts this morning.
As such, Dame Meg wants to know….
-
Why has HM Treasury not designated Amazon Web Services or any other major technology firm as a CTP for the purposes of the Critical Third Parties Regime?
-
How soon can we expect firms to be brought into this regime?
-
There is speculation that the Amazon Web Services outage related to its operations in the United States. Is HM Treasury concerned that seemingly key parts of our IT infrastructure are hosted abroad?
-
HMRC may also have been affected by today’s outage. What work is HM Treasury doing with HMRC to look at what went wrong, and how this may be prevented in future?
AWS: multiple services experienced network connectivity issues
All is not well in the state of Virginia, where it appears today’s disruption is not yet fully fixed.
Amazon Web Services have released two new service updates in the last 25 minutes or so, which appear to admit that the problem is not solved.
AWS say they are investigating issues involving “multiple services” at its US-EAST-1 Region (its date centres based in North Virginia).
It also reports “significant’” errors involving its API services (that’s the Application Programming Interface, which apps use to communicate with each other, and other software).
It said:
-
Oct 20 7:14 AM PDT We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.
-
Oct 20 7:29 AM PDT We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause.
Good news, kids!
As Britain’s children pour out of their schools and head home to their computers, they’ll be reassured to learn that Roblox and Fortnite are both working again.
Having been disrupted by the problems at Amazon Web Services this morning, they both appear to be back to normal.
Roblox reports that “All Systems Operational”, having said early this morning that it was suffering “Widespread outage due to AWS outage”.
There’s good news for Fortnite fans too. Epic Games, its owner, reports that Fortnite, Lego Fortnite, Rocket League and its own game store are all working again.
AWS still experiencing elevated errors
Amazon Web Services has reported it is still suffering “elevated errors” when trying to launch new virtual servers on its cloud computing platform.
In its 13th service update since today’s problems began, AWS says:
We have applied multiple mitigations across multiple Availability Zones (AZs) in US-EAST-1 and are still experiencing elevated errors for new EC2 instance launches. We are rate limiting new instance launches to aid recovery.
Despite today’s AWS outage making headlines worldwide, it doesn’t seem to have alarmed Wall Street.
Shares in Amazon have risen by 0.7% in early trading in New York.
This chart from Downdetector shows how reports of disruption at Amazon Web Services have dropped back after their spike at around 9.30am UK time (4.30am on the East coast of the US).
However, you’ll also note that outage reports are still much higher than before the problems began.
Downdetector: Total global outage reports exceed 6.5 million
Downdetector has revealed that it has received more than 6.5 million reports from users who couldn’t access a web service today.
In the first two hours alone, Downdetector received:
-
Over 1 million reports from the United States
-
More than 400,000 from the United Kingdom
-
Over 200,000 each from Australia, the Netherlands, and Germany
-
Around 180,000 from Japan
“These figures highlight the global scale of the disruption,” it says.
Smart doorbell operator Ring has announced that “a fix has been implemented and Ring services are beginning to see recovery. We will continue to monitor the results.”
Phillip Inman
Amazon cannot process returns of its own parcels, according to at least one Post Office branch in London today, which refused attempts by customers to send back unwanted items this afternoon.
Customers were told that the internet outage meant barcodes used to log returns could not be processed.
It means that the problems experienced by Amazon’s business customers, including Lloyds bank and HM Revenue & Customs, also extends to the host company itself.
Thousands of Amazon parcels are sent back each day by customers and the ease of sending back unwanted items is a popular service, differentiating Amazon from many high street rivals.
All of the phone lines to the Post Office press office rang out when The Guardian called to see how widespread the problem had become.
AWS’s engineers have been working since early this morning on resolving the problems at its Virginia site.
And they report that progress is being made in fixing a problem launching new instances of EC2 (Elastic Compute Cloud), which is a web service that provides virtual servers in the cloud to run applications.
In a new update, AWS say they are now able to successfully launch new EC2 instances in some of its “Availability Zones”, adding:
We are applying similar mitigations to the remaining impacted Availability Zones to restore new instance launches. As we continue to make progress, customers will see an increasing number of successful new EC2 launches.
We continue to recommend that customers launch new EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ.
AWS customers may have “limited recourse” to financial compensation from today’s disruption, warns Ryan Gracey, technology lawyer and partner at law firm Gordons.
Gracey explain:
“AWS customers typically operate under standardised service level agreements, which define uptime commitments and remedies for service failures like today’s outage. While AWS offers service credits for downtime, these are often nominal and do not compensate for losses such as reputational harm or lost revenue. Ultimately, customers will be left with limited recourse.
“The outage reinforces the need for robust contractual negotiation, including tailored SLAs and liability provisions, alongside comprehensive business continuity planning. Organisations must therefore adopt a proactive stance, balancing operational reliance on hyperscale providers with legal safeguards to withstand systemic disruptions.”