Thursday, August 26, 2021

AWS - Dynamo DB Primary Keys, Indexes & Streams

Two types of Primary Keys 

Single Attribute: Single attribute key, called Partition Key or Hash Key. (Unique Id)
Its passed to an internal function that returns the partition name (physical location) where actually item stored.

Composite Key: Multiple attributes key, combination of Partition Key and Sort Key. Partition key decides physical location to store the item and Sort key decide order to store in location.
In this scenario, multiple items could have same partition key with different Sort key.

Two types of Indexes*

Local Secondary Index: 
  • It has same partition key value but different sort key value
  • Can be created only when creating table, can't be added after table creation
  • It can't be deleted
    Ex. User id + his posted threads on a forum 
  • A local secondary index lets you query over a single partition, as specified by the partition key value in the query.
  • When you query a local secondary index, you can choose either eventual consistency or strong consistency.
Global Secondary Index: 
  • It can have different partition key value and different sort key value
  • It can be created with table and can be added later, after table creation
  • It can be deleted
  • A global secondary index lets you query over the entire table, across all partitions.
  • Queries on global secondary indexes support eventual consistency only.
Streams
Stream used to capture any kind of modification in to table data (like CDC in sql)
  • New Item Insert: It capture an image of whole item including all of its attribute 
  • Update on an Item: It captures before and after images of the modified attributes of item.
  • Delete an Item: It capture whole item image before delete.
Stream holds the changes/data for 24 hours then it has deleted from stream.

Stream used with creating triggers for events
A lambda function can be created to trigger events. When ever an insert/update/delete happen lambda function would got triggered.
  • Save data in replica table in another region (DR)
  • Triggering Email for insert/update/delete
    Ex. Send welcome mail to new registered user
Elastic Cache: Elastic cache can be used in conjunction with Dynamo DB to achieve high performance .
Elastic cache provides Redis and Memcahed services.
Query results cached in Elastic cache retrieve faster from application.

Amazon DynamoDB Accelerator (DAX): It is a fully-managed, highly-available, in-memory caching service for DynamoDB.

DAX is a DynamoDB-compatible caching service that enables you to benefit fast in-memory performance for demanding applications. DAX addresses three core scenarios:

1. As an in-memory cache, DAX reduces the response times of eventually-consistent read workloads  from single-digit milliseconds to microseconds.

2. DAX is a service that is API-compatible with Amazon DynamoDB, and thus requires only minimal functional changes to use with an existing application.

3. For read-heavy or bursty workloads, DAX provides increased throughput and potential operational cost savings by reducing the need to over-provision read capacity units. This is especially beneficial for applications that require repeated reads for individual keys.

Capacity Planning: Option available for capacity planning of your DynamoDB Table
If you choose 'On Demand' Read/Write capacity mode, you can not set Read/write throughput and Auto scaling.
Because AWS now manage all for you with some additional charges.


Auto Scaling - Below given are default values if you opt auto scaling for your DynamoDB Table.
on reaching 70% and above it will start to scale 5 to 40000 read/write units.


    

Wednesday, April 7, 2021

AWS - Snowball

Snowball is Amazon's data import/export solution in/out Amazon cloud, in case of large amount of data.


Amazon provides Snowball appliances you load/save data and send it to Amazon to load/save data in cloud. Once data transfer completes & verified Amazon performs software erasure of the appliance to clean the data from device.

Benefits:
It comes with multiple layers of security with 80 and 50 TB capacity.
  • Simple 
  • Fast
  • Secured 
  • Encrypted with AES-256
  • Cost saving (one-fifth of high speed internet)
  • Tamper resistant 
  • Industry standard Trusted Platform Module (TPM): End to end full chain of custody for your data


Types of Snowball
  1. Snowball:  80 and 50 TB storage
  2. Snowball Edge: Same kind of normal Snowball with 100 TB storage + Computation capability
    • Computation capabilities
    • Can run Lambda functions
    • Its a kind of cloud on premise
    • It can create one level of tier between local system and cloud. In case of poor connections/offline on remote locations (offices) system/application can still work   
  3. Snowmobile: Its a 45 foot long ruggedized(shock proof) shipping container with 100 PB data transfer capacity.

It can be used to move massive volumes of data to Amazon. That can include video libraries, images database or even complete data center migration.

Key Points
  1. Snow ball can import data from S3 and export data to S3. If your data stored in Glacier then you need to retrieve data from Glacier to S3 then S3 to Snowball.

Tuesday, April 6, 2021

Weak & Strong Reference

Weak Reference: If we references of some other dlls/assemblies in our program and those assemblies are not signed then the reference to those assembly called Weak Reference.
Why? : A same name assembly with same namespace and class names can be replaced with original one. Our program can't identify the Fake assembly.
If we sign the assembly then our program also check the signed key of the reference assembly.

Strong Reference: Reference a signed assembly in our program is Strong Reference.

Monday, March 8, 2021

AWS - General Design Principals (Serverless)

Take in consideration below given General design principal while designing serverless application

  1. Simple & Singular: Keep functions single purpose and simpler
  2. Design for concurrent requests not total requests: Leverage concurrency  
  3. Orchestrate application with State machine: Use Step functions for big process not with underlying functions chaining
  4. Do not share anything : As lifespan of function is short, you can't rely on storing and sharing in runtime memory of function
  5. Do not write hardware dependent code: As its short lived and get changed 
  6. Design for failure & care for duplicates: For failures make sufficient retries, as well as ensure that it would not generate duplicates   

Saturday, March 6, 2021

AWS - Serverless Layers

 There are 6 layers of serverless application architecture

  • Compute Layer
    • Lambda
    • API Gateway
    • Step functions

  • Data Layer
    • Database
    • S3
    • App sync
      API provides data combining from multiple sources
    • Elastic search 
      Fast search on multiple data sources

  • Messaging & Streaming Layer
    • SNS
    • Kinesis streams (realtime data loading and processing)
    • Kinesis data firehose
      Captures, transforms, and loads streaming data into Kinesis Data Analytics, Amazon S3, Amazon Redshift, and Amazon ES

  • User Management & Identity
    • Cognito
      you can easily add user sign-up, sign-in, and data synchronization to serverless applications. Amazon Cognito user pools provide built-in sign-in screens and federation with Facebook, Google, Amazon, and Security Assertion Markup Language (SAML)

  • Edge Layer
    • Cloud Front

  • System Monitoring & Deployment
    • Cloud watch
    • X-Ray
      Analyze and debug serverless applications by providing distributed tracing and service maps to easily identify performance bottlenecks by visualizing a request end-to-end.
    • Serverless Application Model (AWS SAM)
      An extension of AWS CloudFormation that is used to package, test, and deploy serverless applications. The AWS SAM CLI can also enable faster debugging cycles when developing Lambda functions locally

CI/CD - Safe DB Changes/Migrations

Safe DB Migrations means updating your database schema without breaking the running application and without downtime . In real systems (A...