Hive

Hive:
It organizes data into tables, which provides a means for attaching a structure to data stored in HDFS.
MetaData such as table schema is stored in the a datastore called MetaStore.

Key Points:
1. Data is stored in thetables.
2. Schema is stored in the meta store.
3. Table definitions are local to your machine, so they cannot be shared with other users.
4. The prerequisite for installing the Hive is, the version of the hadoop and the version of the cluster must be same.

To launch hive shell go to the terminal window and type hive and press enter, the prompt will change to hive

hive>

now you are in the hive Environment.

Basically the shell is the way to interact with the Hive. We give commands in this shell. this shell is called HiveQL(Hive Query Language ) its a dialect of SQL. it is influenced by MySQL.

Some basic Commands:
1. hive> show tables;

key points:
1. This command show the tables that are created
2. Hive SQL is a case sensitive. it is only sensitive for the string literals.
3. The database stores its files in a directory called metastore_db, which is relative to the location from which you ran.

to run hive in a non interactive mode, type command as follows
Ex: hive -f file1.q

for short cut scripts, you can use the -e option to specify the commands inline, in which case the final semicolon is not required.

Ex: hive -e 'select * from tab1'

Example:
create a file tab1.txt and write the word hadoop in the text file.
now go to the hive prompt and execute the following Query:

hive -e "create tble tab1(value string);\
              load data local inpath '/tmp/tab1'
               overwrite into table tab1"

Note: Every time when a query is fired, a standard error is printed in the both interactive and non interactive mode such as 'the time taken to run a query'

Example: hive -s -e 'select * from tab1'

Note: in hive we can save the files to the local system and as well as in HDFS.

Example: create table student(sid int,sname,smobile int)  ------------ 1
                 row format delimited                                         ------------ 2
                 fields terminated by '\t';                                     ------------ 3

1. Normal SQL query for creating a table called 'student' with fields id,sname,smobile.
2. Row format clause states that 'each row in the data file is delimited by text'.
3. It expects that there should be 3  fields in each row with types pecified and fields separated by tabs and rows by new lines.

Now we populate data:

Load data local inpath 'input/proc/myex/student.txt' overwrite into table student;

Note: Tables are stored as directories under Hive's warehouse directory, which is controlled by the
  hive.metastore.warehouse.dir property
and Defaults to : /user/hive/warehouse

so the 'student' file is stored in the
/user/hive/warehouse/student/student.txt

Configuration:
hive --config/user/username/dev/hive-conf

Avro

Avro:

its a language neutral-data serialization system.
data is described using a language-independent schema.
code generation is optional here, because reading and writing data that conforms to a  given schema even if your code has not seen that particular schema before. Avro assumes that the schema is always
present at the both read and write time which makes for a very compact encoding, since encoded values do not need to be tagged with a field identifier.

Data Serialization:
in the context of the data storage, serialization is a process of translating data structures or object state into the format such that it can be stored(in the file, or buffer, or transmitted across network connection link) and reconstructed later inthe same or another computer environment.

Avro created to address the downside of the hadoop writables that is language portability.
- it precisely defines the binary format.
- Avro schema's are usually written in JSON and the data usually encoded using a binary format but there are other options too.
- the schema used to read the data need not to be same as the schema used for the write a data

Avro specifies an object container format for sequence of the objects. An Avro data file has a metadata section where the schema is stored which makes the file self-describing. These data files support compression and splittable which is crucial for MapReduce data input format.

Pig, Hive, Crunch, Spark can read and write Avro Data files.
Avro has data types and schema:
Primitive Data types:
null
boolean
int
long
float
double
bytes
string
Complex Data Types:
Array  - An ordered collection of objects. All objects in a particular array must have the same schema

{
  "type" : "array",
  "items":"long"
}

Map: an unordered collection of key value pairs. keys must be strings and values may be any type, although with a particular may all values must have the same schema.
{
  "type": "map",
   "values":"string"
}

Record: collection of the named fields of any type:
{
 "type":"record",
 "name" : "weather record",
 "doc":"a weather reading",
 "fields" : [{"name":"year","type":"int"},
                 {"name":"temperature","type":"int"},
                 {"name":"stationid","type":"string"}
                ]
}

Enum: Aset of named values
{
 "type":"enum",
 "name":"cutlery",
 "doc":"An eating utensil",
 "symbol":["knife","fork","spoon"]
}

Fixed: a fixed number of 8 bits unsigned bytes
{
"type":"fixed",
"name":"marchas",
"size":16
}


Union: a union of the schema's. A union is represented by json array , where each elements in the array is schema. data represented by the union must match one of the schema's in the union.
when a schema is not known ahead of the run time dynamic mapping is used. it is similar to java generic mapping.

Specific mapping or generic mapping:

java has a third mapping , the reflect mapping, which maps Avro types onto persisting java types which uses reflection.
Avro string can be represented by either java string or Avro UTF8 Java type. because UTF8 for efficiency. Single UTF8 instance may be reused for reading or writing a series of the values. Java string decodes UTF-8 object construction time, Avro does it lazily which can increase the performance in same case.

HDFS -- Hadoop Distributed File System.

Distributed File Systems:

files that manage the storage across a network of machines are called Distributed File Systems.
HDFS file system designed for storing a very large files with the streaming data access patters running on a cluster of commodity hardware.

Blocks:
the block size in the hadoop is 128 MB by default these blocks are stored as an independent unit. si if 1 MB file i stored in the HDFS does not use total 128 MB.
HDFS blocks are larger than the disks blocks because to minimize the block sequence.

Name Nodes and Data Nodes:

Name node
 a master node and Data node is a worker nod. there is one name node for a single machine and many data nodes in the same machine. The metadata of the name nodes are managed by the name node. The combination of the name node and data node is called Node. where the map reduce runs the job, slits the input into input splits of block size(by default of 128 MB) and transfers the output of the map function  to the combiner function is feed as the input to the reduce function and final output of the reduce function is stored in the HDFS.

Name node maintains the file system tree and the meta data for all the files and directories in the tree. This information is stored persistently on the local disk in the form of two files
1. the namespace
2. edit log

Data Node
to store and retrive the blocks when they are required.
if name node failes , then the the entire data is lost . for this problem there are two solutions:
1. Backup the files that makeup the persistent state of the file system metadata. Hadoop can be configured so that the name node writes its persistent state to multiple file systems. these writes are the synchronous and atomic.
2. Use secondary name node, the name of the secondary name node may be different but it has the same metadata as that of the first. it is always updated when the original namenode is updated. the secondary name node is stored in the separate machine because it requires more CPU usage as the Original name node is failed, it coopies or starts the secondary name node.

Blocking Caching:
Blocking caching can be done. so that the regular using or accessing the blocks can be catch in the same name node so that other frame works can use the data blocks that are cached.

MapReduce

Map - Reduce:

   Programming model for data processing.

Data Processing :

 operations on data ,m especially by a computer, to retrieve, transform or classify information.

Batch Processing: 

execution of series of programs(jobs) on a computer without any manual intervention. jobs are set up so that they can be run to completion without human interaction.All input parameters are predefined through scripts, command line arguments, control files or job control language.

Hadoop uses parallel processing of data.

Map- Reduce: there are two phases in the Map - Reduce:

The Map Phase and the Reduce Phase. Each phase accepts input and output in key value pairs. so there are two functions

 1. Map Function

 2. Reduce Function.

Map Function extracts the relevant data and then places in the ascending order. Here the keys are ignored, only data is extracted and placed in ascending order. this is the output of the map function.

The output of the map function is used as input of the reduce function. the reduce function reduces the input and the final output is given. the below is the example:

Input to the map function as text format:
(harry,82484Boston)

(harry,298246America)

(albus,23429London)

(albus,23145Britain)

Output of the Map function is :

(harry,Boston)

(harry,America) 

(albus,London)

(albus,Britain)

the output of the map function is a feed to the Reduce function as a input. then the reduce function will produce the output in the r educed format of the data:

(harry,[Boston,America])

(albus,[London,Britain])

Mapper class has four parameters : input key , input value and output key, output value.

Hadoop basic types : 

LongWritable : which corresponds to the Java long.

Text : represents the Java String.

intWritable : represents the Java integer.

Map Method provides an instance called 'Context' to write the output. 

Reduce function also uses the same basic parameters.(input key,input value and output key , output value) important note that the parameter type of the reduce function must be same as map function.

After the reduce function completes its job, the output is saved in a distributed file system. the data here has to split into smaller parts or pieces so that they can be hosted in each machine through YARN (Yet Another Resource Negotiator).

Conclusion:

Map Reduce Job has three important things:

1. Input data

2. Map Reduce Programs

3. Configuration information.

the jobs are divided into tasks: Map task  and Reduce task.

these tasks are scheduled using YARN and run on nodes in the cluster. if a task fails, it will be automatically rescheduled to run on a different node.

here the Hadoop divides the input to a map reduce job into fixed size pieces called input splits or just splits. Hadoop create one map task for each split, which returns the user-defined map function for each record in the split. 

Flow:

1. Input is divided into small pieces  (say same size) and split to map reduce job.

2. Hadoop creates map tasks for each input split such that parallel processing is done.

3. This output is sent to reducer function where it reduces the data.

4. This all is done in nodes

 Data Locality Optimization:

Hadoop does its best to run the map task on a node hwre the input data resides in the HDFS, becaus it doesn't use valuable cluster bandwidth.

Split size should be equal to block size(to remove dominance and overhead).

Some Points To Remember:

Map tasks writes their output to the local disks not to HDFS. Because the output is intermediate output.

This intermediate output is the input for the reduce task which produces the final output. After this the final output is stored in the HDFS files. so the output of the reduce is stored in the HDFS for reliability. For each reduce task of HDFS block of the reduced output , the first replica is stored on the local node, wit the other replicas stored on the off track nodes for reliability.

Map Reduce can be a single reduce task.

it can be a multiple reduce task.

it can be a no reduce task.

Combiner Functions:

Map task produces outputs based on the splits of the input. but to feed the output as the input to the reduce function , the combiner function feeds the outputs of the Map functions as input to the reduce function.

Hadoop Ecosystem

The Core of the Hadoop are:

1. YARN

2. Map-Reduce

3. HDFS.

The following are the elements of  Hadoop Ecosystem:

1. MapReduce.

2. HDFS (Hadoop Distributed File System).

3. YARN (Yet Another Negotiator)

4. Avro

5.Parquet

6. Flume

7. Sqoop

8. Pig

9. Crunch

10. Spark

11. HBase

12. Zookeeper

YARN :

A Resource negotiator, manages the processing resources for Hadoop , schedules jobs, run processing framework.

Map Reduce:

A distributed processing framewok.

Spark: in-memory and streaming processing framework.

HBase: 

NoSQL database  built on HDFS.

Hive: 

SQL processing engine designed for batch processing workloads.

Impala:

 Query engine defined for Business Intelligent workload.

Parquet: 

Very efficient columnar data storage format.

Sqoop:

 Data movement to/from database.

Flume, Kafka : 

Streaming data ingestion.

Solr:

 Powerful text search functionality.

Hue: 

Web based user inteface for the Hadoop.

Sentry:

 Authorization tool provides security for Hadoop.

Need for Hadoop:

Hadoop is used for storage,processing and analyzing the data due to generating more data and retrieving more data by organisations. simple, managing excess or bulk of data.