Senior Data Engineer

slowly changing dimensions

less than 1 minute read

It is for dimension tables where changes are less in source rdbms which we want to get into datawarehouse or hdfs

Type of SCD

SCD Type 1

History is not maintained, data is overwritten.

SCD Type 2

Maintain history. It can be done in below three ways, lets take emaple of product data. All products are stored in products table then:

  • version: maintain multiple version (increasing int value) of same product latest version is the latest details of the product

  • flag: keep flag representing latest details of the product

  • effective date: maintain start date and end date, this way we can keep track of the product details over period of time.

Comments

You May Also Enjoy

spark streaming

9 minute read

In batch processing, at certain frequency batch jobs are run but in case we require that batch to be very very small (depending on requirement, lets say we h...

spark optimizations

9 minute read

optimizations can be at application code level or at cluster level, here we are looking more at cluster level optimizations

spark part-II

9 minute read

spark core works on rdds (spark 1 style) but we have high level constructs to query/process data easily, its dataframe/datasets

hive basics

7 minute read

it is open source datawarehouse to process structured data on top of hadoop