CI/CD - Safe DB Changes/Migrations

Safe DB Migrations means updating your database schema without breaking the running application and without downtime.

In real systems (AWS, Docker, microservices, CI/CD), DB changes are riskier than code changes because data is permanent and shared by multiple services.

Why This Matters in CI/CD & Microservices

• Multiple services share DB
• Rolling deployments mean old + new versions run together
• Containers auto-scale
• Instant rollback must be possible

Unsafe DB change = system outage.

“Safe DB migrations involve backward-compatible schema updates, using the expand-migrate-contract pattern, enabling changes through feature flags, and executing migrations before application rollout to avoid downtime and ensure old and new versions can run simultaneously.”

DB Migration(Keep old schema + new schema both, this will keep old and newly deployed app both working)→ Deploy new App -> Test new app → Enable Feature flag to take new app in effect to users-> delete old schema

1. Backward-Compatible Schema Changes

Goal: Old app version + New app version should both work during deployment.

Bad Example 

DROP COLUMN FirstName;

Old app still expects FirstName → crash.

Good Example 

ALTER TABLE Users ADD COLUMN FullName;

Old app still works, new app starts using new column.

Rule:
Never delete or rename columns immediately.

---

2. Feature Flags

A feature flag lets you deploy DB + code but activate behavior later.

Example flow:

• Add new column FullName
• Deploy code that can use it
• Keep feature OFF
• Turn feature ON after validation

This reduces risk and allows instant rollback without DB rollback.

---

3. Expand → Migrate → Contract Pattern

This is the gold standard for safe DB migrations.

Phase 1 — Expand

Add new schema without removing old.

ALTER TABLE Users ADD COLUMN FullName;

Both old and new apps still work.

---

Phase 2 — Migrate

Move data gradually.

UPDATE Users SET FullName = FirstName + ' ' + LastName;

Background job, no downtime.

---

Phase 3 — Contract

After all services use new column:

DROP COLUMN FirstName;
DROP COLUMN LastName;

Now safe because nothing depends on them.

---

4. Run Migrations Before App Rollout

Correct order:

DB Migration → Deploy App → Enable Feature

Wrong order:

Deploy App → Migration Later → Crash Risk

If app expects a column that doesn’t exist yet → production failure.

---

Ideal Safe Migration Checklist

Practice	Why
Backward compatible schema	Prevent crashes
Expand-Migrate-Contract	Zero downtime
Feature flags	Safe activation
Pre-deployment migrations	Avoid missing schema
Data backfill jobs	Non-blocking
Rollback plan	Disaster recovery

CI/CD - Ideal Pipeline Characteristics

 An Ideal CI/CD Pipeline with Harness means an automated flow where code goes from developer → build → test → security → deploy → monitor with minimum manual work and strong governance.

Think of it as a production assembly line for software.

---

High-Level Flow

Developer → Git Push → CI Build → Test → Security → Artifact → CD Deploy → Monitor → Feedback

Harness mainly shines in Continuous Deployment, but it also supports CI.

“An ideal CI/CD pipeline with Harness automates build, test, security scanning, artifact versioning, progressive deployment strategies like canary or blue-green, governance approvals, and automated rollback with continuous monitoring across environments.”

---

Stage-by-Stage Ideal Pipeline

1. Source Control (Git)

Tools: GitHub / GitLab / Bitbucket

What happens

• Developer pushes code
• Pull request triggers pipeline

Best Practice

• Branch strategy (main / develop / feature)
• Mandatory code review

---

2. Continuous Integration (CI)

Steps

1. Checkout Code
2. Build
3. Unit Tests
4. Code Quality
5. Security Scan
6. Create Artifact (Docker / JAR / DLL)

Typical Tools

• Build: .NET CLI / Maven / npm
• Quality: SonarQube
• Security: Snyk / Trivy
• Artifact Repo: Nexus / Artifactory / ECR

Goal: Ensure every commit is buildable and safe.

---

3. Artifact Storage

Store versioned outputs:

• Docker Images → AWS ECR
• NuGet / JARs → Artifact repo

Key Rule:
Only deploy immutable artifacts, never rebuild in CD.

---

Continuous Deployment (Harness Strength Area)

4. Environment Promotion Flow

Dev → QA → Staging → Prod

Harness provides:

• Visual pipeline
• Rollback automation
• Approval gates
• Feature flags
• Canary / Blue-Green deploy

---

5. Deployment Strategies

Blue-Green

• Old version live
• New version parallel
• Switch traffic instantly

Canary

• 5% → 25% → 50% → 100%
• Auto rollback on failure

Rolling

• Replace containers gradually

Harness automates health checks and rollback without manual SSH.

---

6. Approvals & Governance

Before Prod:

• Manual approval
• Security approval
• Change-management integration
• RBAC controls

This is crucial for enterprises.

---

7. Infrastructure Integration

Works with:

• Kubernetes (EKS)
• ECS / Fargate
• VMs
• Serverless (Lambda)
• Terraform / CloudFormation

Harness doesn’t replace infra tools — it orchestrates them.

---

8. Monitoring & Feedback

After deployment:

• Metrics (CPU, memory, error rate)
• Logs
• APM tools
• Automated rollback triggers

This closes the DevOps loop.

---

Ideal Pipeline Characteristics

Characteristic	Why Important
Automated	Removes human error
Fast	Short feedback cycle
Secure	Scans before deploy
Immutable	Predictable releases
Observable	Quick failure detection
Rollback Ready	Reduces risk
Multi-Env	Dev → Prod consistency

---

Example Ideal Flow (AWS + .NET + Docker)

1. Developer pushes code
2. CI builds Docker image
3. Unit + security tests
4. Push to ECR
5. Harness picks image
6. Deploy to Dev ECS
7. Auto test
8. Promote to QA
9. Manual approval
10. Canary to Prod
11. Auto health check
12. Rollback if error > threshold

CI/CD - Canary Deployment

Below is a clear and practical explanation of how traffic is gradually shifted (“canaried”) in three different environments:

✅ 1. EC2-Based Environment (Classic servers / Auto Scaling Groups)

How Canary Traffic Shifting Works

In EC2-based deployments, canary rollout is typically done at the Load Balancer + Auto Scaling Group (ASG) level.

Step-by-Step Traffic Shift

1. Two ASGs are created

   • ASG1 → Stable version (v1)

   • ASG2 → Canary version (v2)

2. Both ASGs are registered under the same ALB Target Group or separate ALB Target Groups.

3. Weighted Target Groups (ALB Feature)
   Example:

   • v1 TG weight = 90

   • v2 TG weight = 10

4. Gradual shift is done by modifying weights

   • 90/10 → initial canary

   • 70/30 → monitor logs/errors

   • 30/70

   • 0/100 → success

5. If issues appear:

   • instantly push traffic back to v1 by changing weights to 100/0.

Tools commonly used

• AWS ALB Weighted Target Groups

• AWS CodeDeploy (Canary Deployment for EC2/On-Prem)

• ASG + Launch Templates

---

✅ 2. AWS Lambda-Based Environment

Canary deployments are native to Lambda—no infrastructure needed.

How Traffic Shifting Works

Lambda uses Aliases and Versions.

Example:

• Version 1 → Stable

• Version 2 → New release

• Alias “LIVE” points to version 1 initially

Traffic Shift Steps

1. Deploy new Lambda → version 2.

2. Set alias traffic weights:

   LIVE alias:
     v1 = 90%
     v2 = 10%
   

3. Let the system run and monitor logs (CloudWatch).

4. Shift more traffic:

   • v1=70% / v2=30%

   • v1=30% / v2=70%

5. Final cutover:

   • v1=0% / v2=100%

Rollback

Just update the alias back to:

• v1=100%, v2=0%

Tools

• AWS Lambda Aliases + Versions

• AWS SAM / CDK for deployment

• AWS CodeDeploy (Lambda Canary support)

---

✅ 3. Container-Based Environment (ECS / EKS / Kubernetes)

Traffic shifting is done at the service mesh / ingress layer.

---

3A. ECS (EC2 or Fargate)

Traffic Shift Method

• Use two ECS services:

  • service-v1 (old)

  • service-v2 (new)

• Attach services to two ALB Target Groups.

• ALB Weighted Target Groups control traffic:

  • 90/10 → 70/30 → 50/50 → 0/100

Tools

• ALB Weighted Target Groups

• ECS Deployment Controller (Blue/Green via CodeDeploy)

---

3B. EKS / Kubernetes (most common)

How Canary Traffic Shift Works

Canary is controlled by:

• Ingress Controller (NGINX / ALB Ingress)

• Service Mesh (Istio / Linkerd / Consul)

Traffic Shift Example (Istio VirtualService)

spec:
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10

Shift pattern

• 90/10 → Initial Canary

• 70/30 → Monitoring metrics

• 30/70

• 0/100 → Rollout complete

Rollback

Instantly revert weights to:

• v1=100, v2=0

Tools

• Istio / Linkerd / Consul Mesh

• Argo Rollouts (best tool for progressive delivery)

• Kubernetes Ingress with canary annotations

---

🎯 Summary Table

Environment	Mechanism	Traffic Shift Method	Tools
EC2	ASGs + ALB	Weighted Target groups (10% → 100%)	ALB, CodeDeploy
Lambda	Versions + Alias	Alias weight routing	CodeDeploy, SAM, CDK
Containers (ECS)	Services + ALB	Weighted target groups	ALB, ECS B/G deploy
EKS / K8s	Ingress / Mesh	Weighted routing rules	Istio, Argo Rollouts, NGINX Ingress

---

CI/CD - Blue-Green Vs Canary Deployments

 🟦 Blue-Green Deployment

✅ Concept

Blue-Green deployment means you maintain two identical production environments:

• Blue = Current Production
• Green = New Version

Only one is active at a time.

✔ How it works

1. Deploy new application version to the Green environment.

2. Test it thoroughly (Smoke tests, automation, health checks).

3. Switch the traffic from Blue → Green (load balancer flip).

4. If something goes wrong:

   • Switch traffic back to Blue (instant rollback).

---

🟢 Diagram (Simple Flow)

---

⭐ Advantages

Benefit	Description
Zero downtime deployment	Users don't see any interruption.
Instant rollback	One flip back to Blue.
Full environment testing	Green can be validated before traffic shift.

---

⚠️ Disadvantages

Issue	Explanation
Costly	Two full environments required.
Data sync complexity	DB schema changes require careful planning.

---

📌 When to Use Blue-Green

• Releases require zero downtime.
• System is stable, and deployments are less frequent.
• Company can afford 2 parallel production environments.

---

---

🟧 Canary Deployment

✅ Concept

Canary deployment gradually releases the new version to a small subset of users first.

Like in coal mines, a "canary" tests the environment for problems.

✔ How it works

1. Deploy new version v2 to a small percentage (e.g., 1% of users).
2. Monitor behaviors (errors, CPU, logging, metrics).
3. Gradually increase traffic:
• 1% → 5% → 25% → 50% → 100%
4. If issues appear:
• Stop rollout
• Rollback to previous version

---

🟡 Diagram (Traffic Split)

---

⭐ Advantages

Benefit	Description
Very safe	Only a small portion sees new version first.
Real user testing	Validates with live traffic.
Fine-grained rollout control	You can stop at any time.

---

⚠️ Disadvantages

Issue	Description
More complex	Requires intelligent traffic routing & monitoring.
Harder to implement	Needs advanced infra (Istio, Envoy, Feature Flags).
Multiple versions live at once	Must manage compatibility.

---

📌 When to Use Canary Deployment

• High-traffic applications.
• Releases must be very cautious (e.g., banking/fintech).
• Microservices environment.
• Strong observability is in place (Prometheus, Grafana, ELK, AWS CloudWatch).

---

---

🆚 Blue-Green vs Canary — Quick Comparison

Feature	Blue-Green	Canary
Rollback	Instant	Gradual
Complexity	Low	High
Cost	High (2 environments)	Moderate
Risk Level	Medium	Very Low
Traffic Control	All users at once	Small → full rollout
Use Case	Enterprise apps, stable releases	Microservices, critical apps

---

🟩 Real-World Examples (Cloud & Kubernetes)

---

☁️ AWS Example

Blue-Green:

Using AWS Elastic Beanstalk, ALB switch, or CodeDeploy (blue/green).

Canary:

Using API Gateway Canary Release or App Mesh.

---

☸️ Kubernetes Example

Blue-Green:

Deploy new version as deployment-green, switch service selector.

Canary:

Using:

• Istio VirtualService
• Linkerd
• Argo Rollouts
• Flagger

Traffic can be:

• 5% to canary
• 95% to stable

---

🧠 Interview-Ready Summary

🟦 Blue-Green Deployment

➡ Two production environments.
➡ Switch traffic when new version is ready.
➡ Zero downtime + instant rollback.
➡ Easy but expensive.

🟧 Canary Deployment

➡ Slowly roll new version to small % of users.
➡ Real user feedback before full rollout.
➡ Harder but safest for high-risk releases.