hive basics
it is open source datawarehouse to process structured data on top of hadoop
Posts by Tag
- hadoop 6
- hdfs 4
- hive 4
- nosql 4
- database 4
- spark 4
- hbase 3
- scala 3
- compression 2
- jekyll 1
- github 1
- mapreduce 1
- sqoop 1
- csv 1
- xml 1
- json 1
- avro 1
- orc 1
- parquet 1
- optimization 1
- scd 1
- datawarehouse 1
- cap theorem 1
- cassandra 1
- time complexity 1
- data structures 1
- pyspark 1
- python 1
- yarn 1
- spark streaming 1
hadoop
yarn
Yet Another Resource Negotiator Lets first go through how things are in hadoop initial version and what the limitations are which is solved by YARN.
spark part-I
hadoop offers: hdfs: for storage mapreduce: for computation yarn: for resource management
compression in hadoop
Compression will help to: save storage reduce io cost
MapReduce working
MapReduce is programming model for processing big datsets. It consists of two stages: map reduce
hdfs architecture
hdfs is hadoop distributed file system. Highly fault tolerant and is designed to deploy on low cost machines.
hdfs
yarn
Yet Another Resource Negotiator Lets first go through how things are in hadoop initial version and what the limitations are which is solved by YARN.
spark part-I
hadoop offers: hdfs: for storage mapreduce: for computation yarn: for resource management
sqoop working
It is used to transfer data between rdbms to hdfs and vice versa.
hdfs architecture
hdfs is hadoop distributed file system. Highly fault tolerant and is designed to deploy on low cost machines.
hive
hive basics
it is open source datawarehouse to process structured data on top of hadoop
hive for processing and hbase for low latency read
creating table which can be accessed both by hive and hbase, this is done in cases where we require quick (low latency) searches and faster processing of dat...
hive features
hive server / thrift server
hive optimizations
Vectorization
nosql
hive for processing and hbase for low latency read
creating table which can be accessed both by hive and hbase, this is done in cases where we require quick (low latency) searches and faster processing of dat...
cassandra
features
cap theorem
It is for distributed databases. And says that we can have only two out of three gurantees.
hbase
rdbms properties
database
hive for processing and hbase for low latency read
creating table which can be accessed both by hive and hbase, this is done in cases where we require quick (low latency) searches and faster processing of dat...
cassandra
features
cap theorem
It is for distributed databases. And says that we can have only two out of three gurantees.
hbase
rdbms properties
spark
spark optimizations
optimizations can be at application code level or at cluster level, here we are looking more at cluster level optimizations
spark part-II
spark core works on rdds (spark 1 style) but we have high level constructs to query/process data easily, its dataframe/datasets
yarn
Yet Another Resource Negotiator Lets first go through how things are in hadoop initial version and what the limitations are which is solved by YARN.
spark part-I
hadoop offers: hdfs: for storage mapreduce: for computation yarn: for resource management
hbase
hive for processing and hbase for low latency read
creating table which can be accessed both by hive and hbase, this is done in cases where we require quick (low latency) searches and faster processing of dat...
cassandra
features
hbase
rdbms properties
scala
spark part-I
hadoop offers: hdfs: for storage mapreduce: for computation yarn: for resource management
scala part II
scala runs on top of jvm scala is like java so requires main, or we can extends App then we dont have to define main method
scala part I
spark code can be written in different languages (scala, python, java, r), scala is hybrib, oops + functional.
compression
compression in hadoop
Compression will help to: save storage reduce io cost
file formats for big data
There are certain parameters to consider when chossing a file format.
jekyll
Build and deploy static website
To host website we have different ways and for this blog we are focussing on a use case where we need to have a website for blogging, and we are using: je...
github
Build and deploy static website
To host website we have different ways and for this blog we are focussing on a use case where we need to have a website for blogging, and we are using: je...
mapreduce
MapReduce working
MapReduce is programming model for processing big datsets. It consists of two stages: map reduce
sqoop
sqoop working
It is used to transfer data between rdbms to hdfs and vice versa.
csv
file formats for big data
There are certain parameters to consider when chossing a file format.
xml
file formats for big data
There are certain parameters to consider when chossing a file format.
json
file formats for big data
There are certain parameters to consider when chossing a file format.
avro
file formats for big data
There are certain parameters to consider when chossing a file format.
orc
file formats for big data
There are certain parameters to consider when chossing a file format.
parquet
file formats for big data
There are certain parameters to consider when chossing a file format.
optimization
hive optimizations
Vectorization
scd
slowly changing dimensions
It is for dimension tables where changes are less in source rdbms which we want to get into datawarehouse or hdfs
datawarehouse
slowly changing dimensions
It is for dimension tables where changes are less in source rdbms which we want to get into datawarehouse or hdfs
cap theorem
cap theorem
It is for distributed databases. And says that we can have only two out of three gurantees.
cassandra
cassandra
features
time complexity
time complexity
A way to calculate time consumed by an algorithm, as a function of input.
data structures
time complexity
A way to calculate time consumed by an algorithm, as a function of input.
pyspark
spark part-I
hadoop offers: hdfs: for storage mapreduce: for computation yarn: for resource management
python
python basics
module
yarn
yarn
Yet Another Resource Negotiator Lets first go through how things are in hadoop initial version and what the limitations are which is solved by YARN.
spark streaming
spark streaming
In batch processing, at certain frequency batch jobs are run but in case we require that batch to be very very small (depending on requirement, lets say we h...