Index to learn BigData Hadoop Fremework





Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users.It is licensed under the Apache License 2.0.
The Apache Hadoop framework is composed of the following modules:
  • Hadoop Common – contains libraries and utilities needed by other Hadoop modules
  • Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster.
  • Hadoop YARN – a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users' applications.
  • Hadoop MapReduce – a programming model for large scale data processing.
Hadoop HDFS
What is HDFS(Hadoop Distributed File System)? HDFS Architecture, File System, Importing and Exporting Data, listing Files, Creating Directory, Removing Directory, plugins for eclipse, configuration on IDE.

MapReduce
Introduction to MapReduce? Relate MapReduce to Cloud computing, Mapper works, Reducer works,'MapReduce' problems-solutions, Write test MapReduce program, Common MapReduce Algorithms, Sorting and Searching, Indexing, Deeper knowledge of Hadoop API, Using Combiners, Reducing Inter mediate Data with Combiners, Writing Partitioners for Better Load Balancing, Directly Accessing HDFS, Hands-On Exercise.

 
PIG
Introduction to Pig, Where do they fit in? Getting Started with Pig Development, Loading and displaying data, Basic data filters, Pig Schemas, Do some Exercise, PigLatin in-depth, Pig Datatypes, More Advanced Dataset Filtering, Pig Expressions and Functions, Grouping and Sorting Data, Hands-On Exercise, Joining Multiple Datasets,  Validating Datasets, Storing Data, User-Defined Functions, Using functions in Pig, Hands-On Exercise

HIVE 
Introduction to Hive, Hive Architecture, Hive interfaces, Hive architecture, The Hive CLI, Getting data into Hive, Creating tables, Data types, Load data, SerDe, External tables, HiveQL, SQL vs. HiveQL, SELECT/ GROUP BY, Functions/Subqueries, Custom map/reduce scripts,  Joins/Inserting, Hands-on Exercise: Writing queries in HiveQL, Partitioning and Bucketing, Creating partitions, Loading data into partitions, Bucketing, Sampling, Hands-on Exercise: Using partitioning and bucketing, Best Practices for Hive, Configuring Hive, Handling data in Hive, Hands-on Exercise: loading data into Hive,
Importing and Exporting Data from MySQL.

HBASE
What is HBase?(NoSql DataBase), Schema Modeling, The HBase Shell, The HBase Architecture, HBase Java APIs, HBase Data creation using Java Client Programs, Zookeeper.SQOOP
Sqoop Overview, Installation, Imports and Exports,
Importing and Exporting Data Between HDFS and RDBMS(MySQL)

FLUME
Flume Overview, Installation, Import and Export data, Import Streaming Data.

MAHOUT
What is Mahout? What is Machine Learning? Machine Learning Algorithms.
Recommender, Clustering, Classification. User Based Recommender, Item Based Recommender, Hands-on Exercise, Predictive Learning methods.

Comments

Post a Comment

Popular posts from this blog

Setup Nginx as a Reverse Proxy for Thingsboard running on different port/server

In this blog, we will Setup Nginx as a Reverse Proxy for Thingsboard IOT dashboard running on different port Requirement: Ubuntu Server Thingsboard installed on Ubuntu server Nginx installed on Ubuntu server For installing please refer thingsboard documentation page windows:  https://thingsboard.io/docs/user-guide/install/windows/ Linux:  https://thingsboard.io/docs/user-guide/install/linux/ Let us know in comment if you face any issue in installation we will help you. NGINX installation: sudo apt-get install nginx sudo service nginx status sudo service nginx start Open your browser and you will see the default NGinx home page, nginx also runs on default port 80 In some case if you don't see default page up then check: If using cloud machine then need to open "http port 80" on 0.0.0.0/0 or a specific ip If still don't see UI then restart server, sometime it comes after restart. Configure
Read more

How to auto re-launch a YARN Application Master on a failure.

1)Use Case: The fundamental idea of Hadoop2 (Map-Reduce + Yarn) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager ( RM ) and per-application ApplicationMaster ( AM ). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager ( NM ), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. The ResourceManager has two main components: 1. Scheduler: is responsible for allocating resources to the various running applications. 2. Appl
Read more

Read JSON File in Cassandra

Image
INSERT/Load Json data File in Cassandra (3.0+) table Requirement: Create Cassandra table where we can load json data into it. Here some of the column name is separated by space(like 'a b'). Load JSON file into the table. Challenge:  Cassandra support only 'CSV file' load in Table(As per my understanding and searched till now) by using 'COPY' command but not 'JSON' file. Resolution: As per the Cassandra document page, Cassandra supports CSV record level insert and file level insert both operation but for JSON it only support Record level insert by following command: cqlsh> INSERT INTO keyspace1.table1 JSON '{   "id" : "12",    "DB" : "Cassandra",    "ops" : "Insert",   "Project" : "Hadoop" }'; So If we want to insert whole file in the table then we need to loop for each object in JSON file and need to call insert query every tim
Read more