AWS EC2 West outage this morning - well that was fun
by Mountain Computers Inc., Publication Date: Wednesday, December 15, 2021
View Count: 363, Keywords: AWS West, Outage, EC2, Hashtags: #AWSWest #Outage #EC2
at roughly 7am PST, Amazon web services, hosting services, and the related went down.
For me, I am in US West AWS and it was about 30 minutes or so. No worries.
I tested three angles of the problem and got nothing back; so I said, he we go. I wonder how long.
Then, I got a cup of coffee, enjoyed the sunrise; wondered how wide the outage might be, checked the AWS services panel and it said all green; and checked the outage reports and it said 10,000+ reports in 15 minutes, then 14,000+ reports in 30 minutes... then suddenly after 45 minutes... it all came back.
darn. short lived vacation.
now, check your security groups, amis, ec2, monitoring, services, data sizes, file counts, folder counts, md5s, ssls, acls, vpcs, and logs, and see if there is anything fishy. oddly enough, it started about 5am PST and finally collapsed at 7am. monitoring looks for spikes and nothing, not 0. interesting monitoring thought. when a service goes to 0 versus nada, that is a good indicator. i guess i need to add a lower threshold to ensure UCL and LCL (upper and lower control limits). who would have thunk my manufacturing quality control language would come into play here, and stats, and monitoring.
Interior services remained up yet traffic came to an abrupt slow down and then halt. finally, at 7am, something snapped. 45 minutes later all is back as far as one can tell.
backups backup backups. all in the good and green.
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.
© 2023 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.