AWS EC2 West outage this morning - well that was fun

at roughly 7am PST, Amazon web services, hosting services, and the related went down.

 

For me, I am in US West AWS and it was about 30 minutes or so. No worries.

 

I tested three angles of the problem and got nothing back; so I said, he we go. I wonder how long.

 

Then, I got a cup of coffee, enjoyed the sunrise; wondered how wide the outage might be, checked the AWS services panel and it said all green; and checked the outage reports and it said 10,000+ reports in 15 minutes, then 14,000+ reports in 30 minutes... then suddenly after 45 minutes... it all came back.

 

 

 

darn. short lived vacation.

 

now, check your security groups, amis, ec2, monitoring, services, data sizes, file counts, folder counts, md5s, ssls, acls, vpcs, and logs, and see if there is anything fishy. oddly enough, it started about 5am PST and finally collapsed at 7am. monitoring looks for spikes and nothing, not 0. interesting monitoring thought. when a service goes to 0 versus nada, that is a good indicator. i guess i need to add a lower threshold to ensure UCL and LCL (upper and lower control limits). who would have thunk my manufacturing quality control language would come into play here, and stats, and monitoring.

 

Interior services remained up yet traffic came to an abrupt slow down and then halt. finally, at 7am, something snapped. 45 minutes later all is back as far as one can tell.

 

backups backup backups. all in the good and green.

 

REF 1: https://downdetector.com/status/aws-amazon-web-services/

REF 2: https://www.mountaincomputers.org/myBlog/myBlog.asp?mode=view&id=1539  - aws east outage

 

more to come...