AWS Game Day - Scaling, Learning curve - Blue Green Deployment explained

So, after day 2 of the Game Day AWS videos, the concept of Blue-Green Deployment was discussed and I was like what? Oh, you mean cut-over migration and fallback in a cloud environment. I love the new generation of adults where they come up with new terms to describe old situations of a production environment.

 

I must be getting old, I had to develop a Red Team for business recovery and data recovery. I personally like color coding things. Color coding is great yet one still has to test the team for color blind and dyslexia and that is often overcome with good help.

 

Now back to the Blue/Green team consideration.

 

So, Blue means Live and Green means Ready to Deploy and replace Live.

 

I would love to see an enterprise solution that just works without affect 100,000 live clients. Is that possible? Actually, I know of a few clients who do that, and its not usually reported for the success and failures. Some government, some publicly traded, and some private.

 

For me I get it, got it. In all the applications, server, networks, dns, mail, web apps, databases, clusters, caching and deployments I have experienced, the situation is always the same. Do it now, do it fast, and nearly zero latency, zero downtime, zero dark, zero outages, and zero data loss.

 

How can that be achieved? Well, migrating from one to another is easy unless you consider the preponderance of data loss or data gaps, queuing and more.

 

One great example - email - easy, send, fail, queue, retry. Now for other applications like sales orders, financial transactions and settling before deadlines and time sensitive balancing, yes, big worry. Yet what about code in the Blue that changes in the Green that and the underlying data schema changes too?

 

Well, the AWS Game Day folks don't address that. Well, let's do that right now! Let's talk about zero loss of data and migration from one app to another version in a blue green  deployment. How do you do it? I have an answer and do you? let's just say, code has to know its target version and the if the code has a target version that that does not match and is smart enough, will elay and queue the transaction and not error out.

 

That means the application version and the target data source have to have version matching control logic.

 

Scenario: If application A1 is talking to data destination D1 and A1 becomes A2 and A2 needs to talk to D2, and D2 is not available or A1 is talking to do D2 and what have you, what do you do?

 

Version Control verification and matching at scale from the source to destination, ergo client server, etc. I have had version control verification in place for decades, and what one does not want to give away is control to the lower layers in the ISO TCP stack. Keep your control at the application layer where you can modify it without calling support. Remember, downtime is painful and costly.

 

Its called you are Dirty. Queue what you can, Upgrade in Place, and Re-verify data queue and resend.

 

Oh yeah, the roll back plan? What is that in blue/green deployments. It means developers and testers along with production support did not test in their sandbox enough before deployment. I know it sucks, but its life.

 

I have seen major OS releases hung because of a BIOS issue not found in alpha beta and preview. Yet, there it is and Compaq Corporation Server group said, yes, its in about 11,000 data centers and why did you not test against that basic cluster. whoops and pop goes the weasel and monkey. Our upstream product manager got it fixed fast and re-released in like 4 hours... thank you Compaq for great servers in our replication and production release chain, and thank you my product release replication production support team for catching it before it hit the manufacturing release stream and the street. Oh yeah, we extended our release replication engineering production support teams upstream into product channels so we had visibility to release schedules, and 1 to 1 communication contacts for them to us, and us to them. We might not have been in every planning and deployment meeting, yet we were there in spirit, visibility and experience to help with a successful release and successful ongoing support.

 

REF:

https://aws.amazon.com/quickstart/architecture/blue-green-deployment/

https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html

 

more to come...