Distributed File Systems:
files that manage the storage across a network of machines are called Distributed File Systems.
HDFS file system designed for storing a very large files with the streaming data access patters running on a cluster of commodity hardware.
Blocks:
the block size in the hadoop is 128 MB by default these blocks are stored as an independent unit. si if 1 MB file i stored in the HDFS does not use total 128 MB.
HDFS blocks are larger than the disks blocks because to minimize the block sequence.
Name Nodes and Data Nodes:
Name node
a master node and Data node is a worker nod. there is one name node for a single machine and many data nodes in the same machine. The metadata of the name nodes are managed by the name node. The combination of the name node and data node is called Node. where the map reduce runs the job, slits the input into input splits of block size(by default of 128 MB) and transfers the output of the map function to the combiner function is feed as the input to the reduce function and final output of the reduce function is stored in the HDFS.
Name node maintains the file system tree and the meta data for all the files and directories in the tree. This information is stored persistently on the local disk in the form of two files
1. the namespace
2. edit log
Data Node
to store and retrive the blocks when they are required.
if name node failes , then the the entire data is lost . for this problem there are two solutions:
1. Backup the files that makeup the persistent state of the file system metadata. Hadoop can be configured so that the name node writes its persistent state to multiple file systems. these writes are the synchronous and atomic.
2. Use secondary name node, the name of the secondary name node may be different but it has the same metadata as that of the first. it is always updated when the original namenode is updated. the secondary name node is stored in the separate machine because it requires more CPU usage as the Original name node is failed, it coopies or starts the secondary name node.
Blocking Caching:
Blocking caching can be done. so that the regular using or accessing the blocks can be catch in the same name node so that other frame works can use the data blocks that are cached.
files that manage the storage across a network of machines are called Distributed File Systems.
HDFS file system designed for storing a very large files with the streaming data access patters running on a cluster of commodity hardware.
Blocks:
the block size in the hadoop is 128 MB by default these blocks are stored as an independent unit. si if 1 MB file i stored in the HDFS does not use total 128 MB.
HDFS blocks are larger than the disks blocks because to minimize the block sequence.
Name Nodes and Data Nodes:
Name node
a master node and Data node is a worker nod. there is one name node for a single machine and many data nodes in the same machine. The metadata of the name nodes are managed by the name node. The combination of the name node and data node is called Node. where the map reduce runs the job, slits the input into input splits of block size(by default of 128 MB) and transfers the output of the map function to the combiner function is feed as the input to the reduce function and final output of the reduce function is stored in the HDFS.
Name node maintains the file system tree and the meta data for all the files and directories in the tree. This information is stored persistently on the local disk in the form of two files
1. the namespace
2. edit log
Data Node
to store and retrive the blocks when they are required.
if name node failes , then the the entire data is lost . for this problem there are two solutions:
1. Backup the files that makeup the persistent state of the file system metadata. Hadoop can be configured so that the name node writes its persistent state to multiple file systems. these writes are the synchronous and atomic.
2. Use secondary name node, the name of the secondary name node may be different but it has the same metadata as that of the first. it is always updated when the original namenode is updated. the secondary name node is stored in the separate machine because it requires more CPU usage as the Original name node is failed, it coopies or starts the secondary name node.
Blocking Caching:
Blocking caching can be done. so that the regular using or accessing the blocks can be catch in the same name node so that other frame works can use the data blocks that are cached.
No comments:
Post a Comment