CI/CD - Safe DB Changes/Migrations

Safe DB Migrations means updating your database schema without breaking the running application and without downtime.

In real systems (AWS, Docker, microservices, CI/CD), DB changes are riskier than code changes because data is permanent and shared by multiple services.

Why This Matters in CI/CD & Microservices

• Multiple services share DB
• Rolling deployments mean old + new versions run together
• Containers auto-scale
• Instant rollback must be possible

Unsafe DB change = system outage.

“Safe DB migrations involve backward-compatible schema updates, using the expand-migrate-contract pattern, enabling changes through feature flags, and executing migrations before application rollout to avoid downtime and ensure old and new versions can run simultaneously.”

DB Migration(Keep old schema + new schema both, this will keep old and newly deployed app both working)→ Deploy new App -> Test new app → Enable Feature flag to take new app in effect to users-> delete old schema

1. Backward-Compatible Schema Changes

Goal: Old app version + New app version should both work during deployment.

Bad Example 

DROP COLUMN FirstName;

Old app still expects FirstName → crash.

Good Example 

ALTER TABLE Users ADD COLUMN FullName;

Old app still works, new app starts using new column.

Rule:
Never delete or rename columns immediately.

---

2. Feature Flags

A feature flag lets you deploy DB + code but activate behavior later.

Example flow:

• Add new column FullName
• Deploy code that can use it
• Keep feature OFF
• Turn feature ON after validation

This reduces risk and allows instant rollback without DB rollback.

---

3. Expand → Migrate → Contract Pattern

This is the gold standard for safe DB migrations.

Phase 1 — Expand

Add new schema without removing old.

ALTER TABLE Users ADD COLUMN FullName;

Both old and new apps still work.

---

Phase 2 — Migrate

Move data gradually.

UPDATE Users SET FullName = FirstName + ' ' + LastName;

Background job, no downtime.

---

Phase 3 — Contract

After all services use new column:

DROP COLUMN FirstName;
DROP COLUMN LastName;

Now safe because nothing depends on them.

---

4. Run Migrations Before App Rollout

Correct order:

DB Migration → Deploy App → Enable Feature

Wrong order:

Deploy App → Migration Later → Crash Risk

If app expects a column that doesn’t exist yet → production failure.

---

Ideal Safe Migration Checklist

Practice	Why
Backward compatible schema	Prevent crashes
Expand-Migrate-Contract	Zero downtime
Feature flags	Safe activation
Pre-deployment migrations	Avoid missing schema
Data backfill jobs	Non-blocking
Rollback plan	Disaster recovery

CI/CD - Ideal Pipeline Characteristics

 An Ideal CI/CD Pipeline with Harness means an automated flow where code goes from developer → build → test → security → deploy → monitor with minimum manual work and strong governance.

Think of it as a production assembly line for software.

---

High-Level Flow

Developer → Git Push → CI Build → Test → Security → Artifact → CD Deploy → Monitor → Feedback

Harness mainly shines in Continuous Deployment, but it also supports CI.

“An ideal CI/CD pipeline with Harness automates build, test, security scanning, artifact versioning, progressive deployment strategies like canary or blue-green, governance approvals, and automated rollback with continuous monitoring across environments.”

---

Stage-by-Stage Ideal Pipeline

1. Source Control (Git)

Tools: GitHub / GitLab / Bitbucket

What happens

• Developer pushes code
• Pull request triggers pipeline

Best Practice

• Branch strategy (main / develop / feature)
• Mandatory code review

---

2. Continuous Integration (CI)

Steps

1. Checkout Code
2. Build
3. Unit Tests
4. Code Quality
5. Security Scan
6. Create Artifact (Docker / JAR / DLL)

Typical Tools

• Build: .NET CLI / Maven / npm
• Quality: SonarQube
• Security: Snyk / Trivy
• Artifact Repo: Nexus / Artifactory / ECR

Goal: Ensure every commit is buildable and safe.

---

3. Artifact Storage

Store versioned outputs:

• Docker Images → AWS ECR
• NuGet / JARs → Artifact repo

Key Rule:
Only deploy immutable artifacts, never rebuild in CD.

---

Continuous Deployment (Harness Strength Area)

4. Environment Promotion Flow

Dev → QA → Staging → Prod

Harness provides:

• Visual pipeline
• Rollback automation
• Approval gates
• Feature flags
• Canary / Blue-Green deploy

---

5. Deployment Strategies

Blue-Green

• Old version live
• New version parallel
• Switch traffic instantly

Canary

• 5% → 25% → 50% → 100%
• Auto rollback on failure

Rolling

• Replace containers gradually

Harness automates health checks and rollback without manual SSH.

---

6. Approvals & Governance

Before Prod:

• Manual approval
• Security approval
• Change-management integration
• RBAC controls

This is crucial for enterprises.

---

7. Infrastructure Integration

Works with:

• Kubernetes (EKS)
• ECS / Fargate
• VMs
• Serverless (Lambda)
• Terraform / CloudFormation

Harness doesn’t replace infra tools — it orchestrates them.

---

8. Monitoring & Feedback

After deployment:

• Metrics (CPU, memory, error rate)
• Logs
• APM tools
• Automated rollback triggers

This closes the DevOps loop.

---

Ideal Pipeline Characteristics

Characteristic	Why Important
Automated	Removes human error
Fast	Short feedback cycle
Secure	Scans before deploy
Immutable	Predictable releases
Observable	Quick failure detection
Rollback Ready	Reduces risk
Multi-Env	Dev → Prod consistency

---

Example Ideal Flow (AWS + .NET + Docker)

1. Developer pushes code
2. CI builds Docker image
3. Unit + security tests
4. Push to ECR
5. Harness picks image
6. Deploy to Dev ECS
7. Auto test
8. Promote to QA
9. Manual approval
10. Canary to Prod
11. Auto health check
12. Rollback if error > threshold

Docker - Exclude Dev Tools (Docker Optimization)

Do not include development-only tools and packages in the final production container image.

You only keep what is needed to run the app — not what is needed to build or test it.

This reduces image size, vulnerabilities, and startup time, which is very important in AWS environments.

“Removing dev dependencies ensures the production Docker image only contains runtime components, resulting in smaller image size, faster AWS deployments, reduced vulnerabilities, and lower CI/CD and ECR costs.”

How to Achieve: Multi stage docker build 
Create docker file in such a way that makes multi stage build (build then run) 

---

What Are Dev Dependencies?

These are packages/tools used during:

• Local development
• Unit testing
• Debugging
• Building/compiling
• Linting/formatting

But not needed at runtime.

Examples:

Technology	Dev Dependencies
.NET	SDK, test frameworks, analyzers
Node.js	eslint, nodemon, jest
Python	pytest, black
Java	maven, gradle

---

Why Remove Them?

1. Smaller Image

Dev tools are heavy.

Example:

• With SDK + test tools → 1.2 GB
• Runtime only → 200 MB

Smaller image = faster pull from ECR.

---

2. Faster Startup on AWS

Impacts:

• ECS task start
• EKS pod start
• Lambda container cold start

---

3. Better Security

Dev packages increase CVEs.

Removing them:

• Reduces attack surface
• Better ECR / Trivy scan results

---

4. Lower AWS Cost

• Less ECR storage
• Less data transfer
• Faster CI/CD = lower CodeBuild minutes

---

How to Remove Dev Dependencies

.NET Example
Wrong (Includes SDK)

FROM mcr.microsoft.com/dotnet/sdk:8.0

WORKDIR /app

COPY . .

RUN dotnet publish -c Release

ENTRYPOINT ["dotnet", "MyApp.dll"]

SDK stays in final image → huge size.

---

Correct (Multi-Stage)

# Build Stage

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /src

COPY . .

RUN dotnet publish -c Release -o /out

# Runtime Stage

FROM mcr.microsoft.com/dotnet/aspnet:8.0

WORKDIR /app

COPY --from=build /out .

ENTRYPOINT ["dotnet", "MyApp.dll"]

Now:

• SDK removed
• Only runtime DLLs remain

---

Node.js Example

RUN npm install --production

or

RUN npm ci --omit=dev

---

AWS Impact Example

Without removing dev deps:

• Image size: 600 MB
• ECS startup: 25 sec
• Vulnerabilities: 120

With removal:

• Image size: 150 MB
• ECS startup: 6–8 sec
• Vulnerabilities: 30

---

Simple Analogy

It’s like shipping a car:

• Dev deps = factory tools, welding machines
• Runtime deps = engine, wheels, fuel

Customer only needs the car, not the factory.

Docker - Layer Caching

Layer Caching in Docker means Docker reuses previously built image layers instead of rebuilding everything from scratch.

In AWS + .NET context, this is very important for CI/CD speed, cost, and developer productivity.

“Docker layer caching improves AWS CI/CD performance by reusing unchanged build steps. In .NET, copying the .csproj and running dotnet restore before copying source code prevents unnecessary dependency restores, reducing build time, ECR push size, and deployment latency.”

---

First Understand Docker Layers

Every Dockerfile instruction creates a layer:

FROM ...
WORKDIR ...
COPY ...
RUN ...

Docker stores each layer as a snapshot.

If nothing changes in that step → Docker reuses the cached layer.

---

Why It Matters on AWS

Layer caching helps in:

AWS Service	Benefit
CodeBuild	Faster builds
ECR	Smaller push size
ECS/EKS	Faster deployments
Lambda Containers	Reduced cold start
CI/CD Pipelines	Minutes saved per build

---

Example Without Layer Optimization (.NET)

FROM mcr.microsoft.com/dotnet/sdk:8.0
WORKDIR /app
COPY . .
RUN dotnet restore
RUN dotnet publish -c Release -o out

Problem:

• Any small code change invalidates COPY . .
• Docker reruns restore + publish
• Slow builds (2–5 minutes)

---

Optimized Version Using Layer Caching

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /app

# Copy only project files first
COPY *.csproj .
RUN dotnet restore

# Then copy rest of source
COPY . .
RUN dotnet publish -c Release -o out

Why This Works

• .csproj changes rarely
• Source code changes frequently

Docker cache logic:

Step	Changes Often?	Cache Used?
COPY *.csproj	Rare	Yes
RUN dotnet restore	Rare	Yes
COPY . .	Often	No
RUN publish	Often	No

So NuGet restore is skipped most builds, saving time.

---

AWS CI/CD Impact Example

Without caching:

• Build time = 6 minutes
• Every commit full rebuild

With caching:

• Build time = 1.5–2 minutes
• Only changed layers rebuilt

In CodeBuild or GitHub Actions, this saves hours per week.

---

ECR Push Optimization

Docker only pushes changed layers.

If only app code changed:

• Push size maybe 20–30 MB
• Instead of 300–500 MB

Less network → faster deploy → lower cost.

---

Real-World Analogy

Think of it like Excel recalculation:

• Change one cell → not entire sheet recalculated
• Docker only rebuilds changed steps

---

Best Practices for Layer Caching (.NET + AWS)

1. Order Matters

Stable steps first, changing steps later.

2. Separate Restore

COPY *.csproj .
RUN dotnet restore

3. Use .dockerignore

Exclude:

• bin/
• obj/
• .git/
• logs/

Reduces invalid cache triggers.

4. Multi-Stage Build

Keeps final image small and cache efficient.

AWS - Linux Alpine images

Using Alpine Linux as the operating system layer of your Docker image instead of a full Linux distro like Ubuntu or Debian.

It is not AWS-specific — but it is very popular on AWS because of performance and cost benefits.

“Alpine base images on AWS reduce container size, improve startup time, lower ECR costs, and reduce vulnerabilities, but require compatibility testing due to musl vs glibc differences.”

---

What is Alpine Linux?

Alpine Linux is a very small, security-focused Linux distribution.

Typical sizes:

Base Image	Approx Size
Ubuntu	70–120 MB
Debian	60–100 MB
Alpine	5–15 MB

So your container becomes much smaller.

---

Example in Dockerfile (.NET)

Normal:

FROM mcr.microsoft.com/dotnet/aspnet:8.0

Alpine:

FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine

Sample Alpine Docker file

FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine

WORKDIR /app

COPY --from=build /app/out .

ENTRYPOINT ["dotnet", "MyApp.dll"]

Difference:

• Same .NET runtime
• Smaller OS layer

---

Why It Matters on AWS

1. Faster Image Pull (ECR → ECS/EKS/Lambda)

Smaller image = faster download.

Example:

• Ubuntu image: 400 MB → 15–20 sec pull
• Alpine image: 90 MB → 3–5 sec pull

This directly impacts:

• Lambda cold start
• ECS task startup
• Auto-scaling speed

---

2. Lower Storage Cost (ECR)

AWS ECR charges for storage.

Smaller image:

• Less GB stored
• Lower monthly cost

---

3. Better Security

Alpine includes:

• Fewer packages
• Smaller attack surface
• Fewer CVEs

Security scans (ECR / Trivy) usually show less vulnerabilities.

---

4. Faster CI/CD Pipelines

In CodeBuild / GitHub Actions:

• Faster build
• Faster push/pull
• Less network usage

---

Where You Use It on AWS

AWS Service	Benefit
ECS Fargate	Faster container start
EKS (Kubernetes)	Faster pod scheduling
Lambda Container	Reduced cold start
CodeBuild	Faster pipelines
ECR	Lower storage cost

---

Important Caveat (Very Important)

Alpine uses musl libc instead of glibc.

Some libraries or native dependencies may fail, especially:

• Image processing libs
• Oracle drivers
• Some older .NET native packages
• Python scientific libs

If your app depends on native binaries, test carefully.

---

When NOT to Use Alpine

Avoid Alpine if:

• You need heavy native libraries
• You see runtime crashes related to libc
• Vendor software requires glibc
• You need full debugging tools

In such cases use:

• -slim images
• Debian slim
• Ubuntu minimal

---

AWS - Multi-Stage Docker Build

Multi stage sample docker file

# Build stage

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /app

COPY . .

RUN dotnet publish -c Release -o out

# Runtime stage

FROM mcr.microsoft.com/dotnet/aspnet:8.0

WORKDIR /app

COPY --from=build /app/out .

ENTRYPOINT ["dotnet", "MyApp.dll"]

This Dockerfile uses Multi-Stage Build:

• Stage 1 → Build the app
• Stage 2 → Run the app

The RUN command executes during image build time, not when the container starts.

---

Stage 1 – Build Stage

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

• Pulls Microsoft .NET SDK image
• SDK contains compiler, NuGet, build tools
• Named this stage build

---

WORKDIR /app

• Sets working directory inside container to /app
• All next commands run from here

---

COPY . .

• Copies your entire project folder from local machine → container /app
• Includes .csproj, source code, etc.

---

RUN dotnet publish -c Release -o out

This is the key line 

RUN executes a command while building the image.

Think of it as:

“Open a temporary container, run this command, save the result as a new image layer.”

What happens internally

1. Docker creates a temporary container from sdk:8.0
2. Runs:
• dotnet publish -c Release -o out

3. .NET does:

• Restore NuGet packages
• Compile code
• Optimize for Release
• Output DLLs + dependencies into /app/out
4. Docker commits the result as a new image layer
5. Temporary container is deleted

So RUN is build-time execution, not runtime.

Result now inside image:

/app/out/MyApp.dll
/app/out/*.json
/app/out/*.deps.json

---

Stage 2 – Runtime Stage

FROM mcr.microsoft.com/dotnet/aspnet:8.0

• Starts a fresh lightweight image
• No SDK, only runtime
• Much smaller

---

WORKDIR /app

Sets working directory again.

---

COPY --from=build /app/out .

• Copies only compiled output from build stage
• Not source code
• Not SDK
• Only /app/out → /app

This is why image becomes small.

---

ENTRYPOINT ["dotnet", "MyApp.dll"]

This runs when container starts, not during build.

Difference:

Instruction	When it runs
RUN	During docker build
ENTRYPOINT	During docker run

---

Lifecycle Summary
When you run:

docker build -t myapp .

Docker executes:

1. Pull SDK image
2. Copy code
3. RUN dotnet publish → compile app
4. Start new runtime image
5. Copy compiled files
6. Create final image

---

When you run:

docker run myapp

Only this runs:

dotnet MyApp.dll

No compiling happens here.

---

Simple Analogy

• RUN = “Bake the cake in the kitchen”
• ENTRYPOINT = “Serve the cake to the customer”

RUN prepares the app.
ENTRYPOINT executes the app.

---

Why This Is Efficient

Without multi-stage:

• Image ~1–2 GB
• Includes compiler

With multi-stage:

• Image ~150–250 MB
• Only runtime files
• Faster deploys
• More secure

So RUN = build-time command that produces artifacts saved into the image layer.