Harness | Core Concepts

 Core components of Harness

1. Application

An Application is the top-level container in Harness.

• It represents a complete project or system
• Contains all services, environments, workflows, and pipelines for that project

Example

Application: E-commerce Platform

Inside it you may have:

• payment-service
• order-service
• notification-service

---

2. Service

A Service represents a deployable microservice or application component.

It defines:

• Artifacts (Docker image, JAR, ZIP, etc.)
• Service configuration
• Deployment variables

Example

Service: Order Service

Artifacts:

• Docker image from Docker
• JAR file from a repository
• Lambda package

Possible service types include:

• Secure Shell
• Kubernetes
• AWS Lambda
• ECS

Example:

• If deploying a container to Kubernetes → Kubernetes service type
• If deploying serverless → use AWS Lambda service type.

---

3. Environment

An Environment represents where the service will be deployed.

Typical environments:

• Dev
• QA
• Staging
• Production

It defines:

• Infrastructure
• Cloud provider
• Target cluster or server

Example:
Environment: Production

Infrastructure:

• Kubernetes cluster
• AWS account
• VM instances

---

4. Workflow

Workflows are part of First Gen Harness in Next Gen Harness its moved under Pipelines Execution.

A Workflow defines how the deployment happens.

It contains:

• Deployment steps
• Pre-deployment tasks
• Post-deployment verification
• Rollback steps

Example workflow steps:

1. Pull artifact
2. Deploy container
3. Run database migration
4. Perform health check
5. Verify deployment

Harness can also integrate with monitoring tools for verification.

---

5. Pipeline

A Pipeline orchestrates multiple workflows.

It allows:

• Sequential deployments
• Multi-environment promotion
• Approval gates

Example pipeline:

Build → Dev Deploy → QA Deploy → Approval → Production Deploy

Example integration:

• Build triggered from Jenkins
• Deploy to Kubernetes cluster

---

Complete Example

Application

Online Banking System

Services

• account-service
• transaction-service
• notification-service

Environments

• Dev
• QA
• Production

Workflow

Deploy container → Run health check → Verify logs

Pipeline

Build → Dev → QA → Manual Approval → Production

---

Easy way to remember

• Application → Project
• Service → What you deploy
• Environment → Where you deploy
• Workflow → How you deploy
• Pipeline → Deployment process across environments

AWS | Using Interface Endpoint Vs Arn

Interface Endpoint is used to connect to services like SQS, SNS, Secret Manager, ECR CloudWatch, but these services can be accessed directly through their ARNs, then what is the benefit of Interface Endpoint? 

Key Clarification

ARN controls authorization
VPC Endpoint controls network routing

These are completely different layers.

---

Without Interface Endpoint

Let’s say your EC2 instance calls SQS:

EC2 → sqs.us-east-1.amazonaws.com

Even if:

• You use the correct ARN
• IAM permissions are correct

The traffic still:

1. Leaves your VPC

2. Goes through:

• Internet Gateway OR
• NAT Gateway
3. Travels over public AWS endpoints

Even though it's inside AWS backbone, it's still using public endpoints.

---

With Interface Endpoint

Now:

EC2 → Private ENI inside VPC → SQS

Traffic:

• Never goes to internet
• Never needs NAT Gateway
• Never needs public IP
• Stays fully private

---

So What's the Real Benefit?

No Internet Required

If your subnet is fully private:

• No IGW
• No NAT
• No public IP

Your EC2 cannot reach service SQS/SNS/SSM etc. at all without:

• NAT Gateway + IGW OR
• Interface Endpoint (Without NAT and IGW)

So the Interface endpoint enables connectivity in fully private networks.

---

Stronger Security Posture

Without endpoint:

• Your subnet must have outbound internet access.

With endpoint:

• You can remove NAT completely.
• Block all outbound internet traffic.
• Only allow traffic to specific AWS services.

Much tighter security boundary.

---

Endpoint Policy Control (Very Powerful)

Interface endpoints support endpoint policies.

Example:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

Now even if IAM allows more, the endpoint itself restricts what can pass through.

This gives network-level authorization filtering.

You cannot do this with just ARN + IAM alone.

---

Compliance & Regulated Environments

In regulated industries (banking, healthcare):

Rules may require:

• No internet connectivity
• No public endpoints
• Private routing only

Interface endpoints satisfy those requirements.

---

NAT Gateway Cost Optimization

NAT Gateway is expensive.

If you only need AWS service access (SQS, SSM, SNS, etc.):

You can:

• Remove NAT
• Use interface endpoints

For high-traffic systems, this can save serious money.

---

When You DON’T Need Interface Endpoint

If:

• Your subnet already has NAT
• You are OK with outbound internet access
• No strict compliance requirements
• Low security sensitivity

Then you may not need it.

Small projects often skip it.

---

Real Production Example

Private EKS cluster:

• No public subnets
• No NAT
• No outbound internet allowed

Pods need to:

• Pull images from ECR
• Read secrets
• Send SQS messages
• Register with SSM

Without interface endpoints?

❌ Not possible.

With interface endpoints?

✅ Fully private operation.

---

Final Answer

You don't use interface endpoints for permission.

You use them for:

• Private networking
• Security isolation
• Compliance
• Removing internet dependency
• Reducing NAT usage

Onion / Clean Architecture Deep Dive

 Both Clean Architecture and Onion Architecture follow the same core rule:

🔑 Dependencies always point inward toward the Domain.

---

Onion / Clean Architecture – Layer Overview

Typical layers (from inside to outside):

[ Domain ]  ← pure business logic
[ Application ]  ← use cases
[ Infrastructure ]  ← database, external services
[ Presentation ]  ← controllers, API, UI

---

Example: Employee Management System

We’ll implement:

“Create a new Employee and save it.”

---

Domain Layer (Core Business Logic)

📁 Domain/

This is the center of the onion.

Where to define domain objects?

👉 HERE.

Domain objects like:

• Employee

• Order

• Product

• Customer

must live in Domain.

They must NOT depend on:

• Database

• Frameworks

• Controllers

• UI

---

Domain Entity

📁 Domain/Entities/Employee.cs

namespace Domain.Entities
{
    public class Employee
    {
        public Guid Id { get; private set; }
        public string Name { get; private set; }
        public decimal Salary { get; private set; }

        public Employee(string name, decimal salary)
        {
            if (salary <= 0)
                throw new ArgumentException("Salary must be greater than zero");

            Id = Guid.NewGuid();
            Name = name;
            Salary = salary;
        }

        public void IncreaseSalary(decimal amount)
        {
            if (amount <= 0)
                throw new ArgumentException("Increase must be positive");

            Salary += amount;
        }
    }
}

✔ Business rules live here
✔ No EF, no DB, no HTTP

---

Repository Interface (Contract)

📁 Domain/Interfaces/IEmployeeRepository.cs

namespace Domain.Interfaces
{
    public interface IEmployeeRepository
    {
        void Add(Employee employee);
        Employee GetById(Guid id);
    }
}

IMPORTANT:

The interface lives in Domain, not Infrastructure.

Why?

Because Domain defines what it needs.
Infrastructure will implement it.

This keeps dependencies inward.

---

Application Layer (Use Cases)

📁 Application/

This layer orchestrates business operations.

Where to implement use cases?

👉 HERE.

Example use case:

• CreateEmployee

• IncreaseSalary

• GetEmployee

---

Create Employee Use Case

📁 Application/UseCases/CreateEmployee.cs

using Domain.Entities;
using Domain.Interfaces;

namespace Application.UseCases
{
    public class CreateEmployee
    {
        private readonly IEmployeeRepository _repository;

        public CreateEmployee(IEmployeeRepository repository)
        {
            _repository = repository;
        }

        public Guid Execute(string name, decimal salary)
        {
            var employee = new Employee(name, salary);
            _repository.Add(employee);

            return employee.Id;
        }
    }
}

✔ Uses Domain entity
✔ Uses Domain interface
✔ Does NOT know about database

---

Infrastructure Layer (External World)

📁 Infrastructure/

This is where:

• EF Core

• SQL

• MongoDB

• File system

• External APIs

live.

---

Where to implement repository?

👉 HERE.

---

EF Repository Implementation

📁 Infrastructure/Repositories/EmployeeRepository.cs

using Domain.Entities;
using Domain.Interfaces;

namespace Infrastructure.Repositories
{
    public class EmployeeRepository : IEmployeeRepository
    {
        private readonly AppDbContext _context;

        public EmployeeRepository(AppDbContext context)
        {
            _context = context;
        }

        public void Add(Employee employee)
        {
            _context.Employees.Add(employee);
            _context.SaveChanges();
        }

        public Employee GetById(Guid id)
        {
            return _context.Employees.Find(id);
        }
    }
}

✔ Implements Domain interface
✔ Depends on EF
✔ Domain does NOT depend on this

Dependency direction:

Infrastructure → Domain

Correct ✅

---

Presentation Layer (API / UI)

📁 WebAPI/Controllers/EmployeeController.cs

---

Where to call the use case?

👉 HERE (Presentation Layer)

---

using Application.UseCases;
using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/employees")]
public class EmployeeController : ControllerBase
{
    private readonly CreateEmployee _createEmployee;

    public EmployeeController(CreateEmployee createEmployee)
    {
        _createEmployee = createEmployee;
    }

    [HttpPost]
    public IActionResult Create(string name, decimal salary)
    {
        var id = _createEmployee.Execute(name, salary);
        return Ok(id);
    }
}

✔ Controller calls Use Case
✔ Controller does NOT create Employee directly
✔ Controller does NOT talk to DB

---

Summary – Clear Answers
❓ Where to define domain objects like Employee or Order?

Domain Layer

Domain/Entities/Employee.cs
Domain/Entities/Order.cs

They contain:

• Business rules
• Validation
• Core behavior

They must NOT know:

• Database
• UI
• Frameworks

---

❓ Where to implement repository?

• Interface → Domain
• Implementation → Infrastructure

Domain/Interfaces/IEmployeeRepository
Infrastructure/Repositories/EmployeeRepository

---

❓ Where to implement business logic?

• Core rules → Domain
• Use case orchestration → Application

---

❓ Where to call use cases?

From:

• Controllers
• API endpoints
• UI
• Background jobs

That’s the Presentation Layer.

---

Dependency Rule (Most Important)

Only this direction is allowed:

Presentation → Application → Domain
Infrastructure → Domain

🚫 Domain must never depend on outer layers.

---

Clean vs Onion?

Practically identical in structure.

• Onion Architecture was popularized by Jeffrey Palermo
• Clean Architecture was formalized by Robert C. Martin

Both enforce:

“Business rules should not depend on frameworks.”

AWS | Trusted Advisor

An advisor service that scans your resources and infrastructure compare them with best practices and provide recommendations on important aspects like cost, performance, security and fault tolerance.

AWS | Athena Vs S3 Select Vs Redshift Spectrum

Comparison between Athena, S3 Select and Redshift Spectrum

Feature	Amazon S3 Select	Amazon Athena	Amazon Redshift Spectrum
What it is	Query individual objects in S3	Serverless SQL query engine on S3	Query S3 from Redshift
Best for	App-level filtering of single files	Ad-hoc analytics	Enterprise data warehouse extension
Setup	None (API call)	None (serverless)	Requires Redshift cluster
SQL support	Limited (simple SQL)	Full ANSI SQL	Full Redshift SQL
Performance	Good for small object filtering	Good for medium-large datasets	Best for very large datasets
Pricing model	Per data scanned	Per TB scanned	Per TB scanned + Redshift cost
Concurrency	App-controlled	High	Very high
Use case example	Fetch specific rows from JSON/CSV in app	Run analytics on data lake	Join S3 data with warehouse tables

---

When to Use What

Use S3 Select when:

• You need to retrieve specific rows from one object
• You're inside an application
• You want to reduce data transfer

Think: “Filter before downloading.”

---

Use Athena when:

• You have a data lake in S3
• You want SQL without managing infrastructure
• You need BI / analytics

Think: “Serverless analytics on S3.”

---

Use Redshift Spectrum when:

• You already use Redshift
• You want to join warehouse tables + S3 data
• You need enterprise-scale performance

Think: “Extend data warehouse to S3.”

---

Simple Decision Rule

• Single file → S3 Select
• Data lake analytics → Athena
• Enterprise warehouse + S3 → Redshift Spectrum

Athena Vs S3 Select

Amazon Athena

A serverless interactive query service that lets you run full SQL queries directly on data stored in S3.

Amazon S3 Select

A feature of S3 that lets you retrieve only a subset of data from a single object using simple SQL expressions.

---

Key Differences

Feature	Athena	S3 Select
Scope	Query across multiple files	Query within a single object
SQL Support	Full ANSI SQL	Limited SQL (simple SELECT, WHERE)
Use Case	Analytics, reporting, BI	Efficient object-level filtering
Performance	Scans full dataset (optimized by partitioning)	Reads only selected data from object
Pricing	Per TB scanned	Per GB scanned + data returned
Schema	Requires table definition (Glue/Data Catalog)	No external catalog needed
Joins	Yes	No
Aggregations	Yes	Very limited

---

When to Use Each

Use Athena when:

• You need to query large datasets across many files
• You need joins, aggregations, grouping
• You're connecting BI tools (e.g., QuickSight)
• You want serverless analytics without managing infrastructure

Example:

SELECT customer_id, SUM(amount)
FROM transactions
GROUP BY customer_id;

---

Use S3 Select when:

• You need to fetch a small portion of a single large file
• You want to reduce network transfer
• You’re building an application that reads filtered object data
• You need low-latency object-level filtering

Example:

SELECT * FROM s3object s WHERE s.status = 'active'

---

Cost Consideration

• Athena can become expensive if queries scan large unpartitioned datasets.
• S3 Select is often cheaper when extracting small pieces of large objects.

---

Simple Mental Model

• Athena = Data warehouse-style querying over S3
• S3 Select = Smart “grep” inside one S3 file

Web API - Caching

Note: Response/Output cache handled by their respective middlewares and do not hit controller in subsequent requests until cache invalidated.

Data Cache (IMemorycache, Memcache, Redis): Managed by code, you check in code if data available in cache then serve else reproduce and serve.

1. Output Caching (Recommended for .NET 7+)

Starting in ASP.NET Core 7+, Microsoft introduced built-in Output Caching Middleware, which is the best way to cache full API responses.

Step 1: Register Output Caching

In Program.cs:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOutputCache();

var app = builder.Build();

app.UseOutputCache();

app.MapControllers();

app.Run();

---

Step 2: Apply Caching to an Endpoint

[HttpGet]
[OutputCache(Duration = 60)]
public IActionResult GetProducts()
{
    return Ok(GetProductsFromDatabase());
}

🔹 This caches the response for 60 seconds.
🔹 Works great for GET endpoints.

---

Advanced Example (Custom Policy)

builder.Services.AddOutputCache(options =>
{
    options.AddPolicy("MyPolicy", policy =>
        policy.Expire(TimeSpan.FromMinutes(5))
              .SetVaryByQuery("category"));
});

Then:

[OutputCache(PolicyName = "MyPolicy")]

---

2. Response Caching (HTTP Cache Headers)

This uses HTTP headers like Cache-Control. It works with browser/proxy caching.

Enable Middleware

builder.Services.AddResponseCaching();
app.UseResponseCaching();

Use Attribute

[ResponseCache(Duration = 60)]
[HttpGet]
public IActionResult GetProducts()
{
    return Ok(data);
}

🔹 This sets HTTP headers.
🔹 It does NOT cache on the server by default.
🔹 Good for public APIs.

---

3. Data Cache 
In-Memory Caching (IMemoryCache)

Best when you want to cache data, not full HTTP responses.

Register

builder.Services.AddMemoryCache();

Use in Controller

private readonly IMemoryCache _cache;

public ProductsController(IMemoryCache cache)
{
    _cache = cache;
}

[HttpGet]
public IActionResult GetProducts()
{
    if (!_cache.TryGetValue("products", out List<Product> products))
    {
        products = GetProductsFromDatabase();

        var cacheOptions = new MemoryCacheEntryOptions()
            .SetSlidingExpiration(TimeSpan.FromMinutes(5));

        _cache.Set("products", products, cacheOptions);
    }

    return Ok(products);
}

---

4. Distributed Caching (Redis, SQL Server)

For multi-server environments, use distributed cache.

Common choice:

• Redis: Redis can store complex data types like Lists, arrays, Hash tables etc.

• Memcached: Simple text string store 

Example Setup with Redis

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "localhost:6379";
});

Then inject IDistributedCache.

Feature	Memcached	Redis
Distributed	Yes	Yes
Replication	No (basic)	Yes
Persistence	No	Yes
Data loss risk	Higher	Lower
Complexity	Simple	Advanced

---

Which One Should You Use?

Scenario	Recommended
Cache full API response	OutputCache
Scale across multiple servers	Redis (Distributed Cache)
Cache DB results only	IMemoryCache
Browser/proxy caching	ResponseCache

---

Best Practice Tips

• Cache only GET requests.
• Avoid caching user-specific data unless using VaryBy.
• Always define expiration.
• Invalidate cache on data update.
• Use distributed cache in production with multiple instances.

Output cache and Response Cache both stores whole response handled by middleware, only difference is storage location. 

Response Caching vs Output Caching

Feature	Response Caching	Output Caching
Where caching happens	Client / Proxy (browser, CDN)	Server (inside ASP.NET Core)
Middleware	UseResponseCaching()	UseOutputCache()
Actually stores response on server?	No	Yes
Performance benefit for server	Minimal	Significant
Best for	Public APIs	High-performance APIs
Introduced	Early ASP.NET Core versions	.NET 7

CI/CD - Safe DB Changes/Migrations

Safe DB Migrations means updating your database schema without breaking the running application and without downtime.

In real systems (AWS, Docker, microservices, CI/CD), DB changes are riskier than code changes because data is permanent and shared by multiple services.

Why This Matters in CI/CD & Microservices

• Multiple services share DB
• Rolling deployments mean old + new versions run together
• Containers auto-scale
• Instant rollback must be possible

Unsafe DB change = system outage.

“Safe DB migrations involve backward-compatible schema updates, using the expand-migrate-contract pattern, enabling changes through feature flags, and executing migrations before application rollout to avoid downtime and ensure old and new versions can run simultaneously.”

DB Migration(Keep old schema + new schema both, this will keep old and newly deployed app both working)→ Deploy new App -> Test new app → Enable Feature flag to take new app in effect to users-> delete old schema

1. Backward-Compatible Schema Changes

Goal: Old app version + New app version should both work during deployment.

Bad Example 

DROP COLUMN FirstName;

Old app still expects FirstName → crash.

Good Example 

ALTER TABLE Users ADD COLUMN FullName;

Old app still works, new app starts using new column.

Rule:
Never delete or rename columns immediately.

---

2. Feature Flags

A feature flag lets you deploy DB + code but activate behavior later.

Example flow:

• Add new column FullName
• Deploy code that can use it
• Keep feature OFF
• Turn feature ON after validation

This reduces risk and allows instant rollback without DB rollback.

---

3. Expand → Migrate → Contract Pattern

This is the gold standard for safe DB migrations.

Phase 1 — Expand

Add new schema without removing old.

ALTER TABLE Users ADD COLUMN FullName;

Both old and new apps still work.

---

Phase 2 — Migrate

Move data gradually.

UPDATE Users SET FullName = FirstName + ' ' + LastName;

Background job, no downtime.

---

Phase 3 — Contract

After all services use new column:

DROP COLUMN FirstName;
DROP COLUMN LastName;

Now safe because nothing depends on them.

---

4. Run Migrations Before App Rollout

Correct order:

DB Migration → Deploy App → Enable Feature

Wrong order:

Deploy App → Migration Later → Crash Risk

If app expects a column that doesn’t exist yet → production failure.

---

Ideal Safe Migration Checklist

Practice	Why
Backward compatible schema	Prevent crashes
Expand-Migrate-Contract	Zero downtime
Feature flags	Safe activation
Pre-deployment migrations	Avoid missing schema
Data backfill jobs	Non-blocking
Rollback plan	Disaster recovery